Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
6.5.0
-
6.5.0-3875
-
Untriaged
-
Unknown
Description
Steps to reproduce:
1. Initialize a 2 node cluster with default bucket having replica=1
+----------------+----------------------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------------------------+--------------+
|
| 172.23.104.202 | [u'index', u'kv', u'n1ql'] | Cluster node |
|
| 172.23.104.203 | None | <--- IN --- |
|
+----------------+----------------------------+--------------+
|
2. Loaded 50k items into the bucket with durability=majority
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
| Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
|
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
| default | membase | 1 | 0 | 50000 | 3961520128 | 153073520 | 196070128 |
|
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
3. Rebalance in 2 nodes and load 25k more items in parallel with durability=majority. Lots of DurabilityImpossibleExceptions are seen which is expected as per. Rebalance completed properly. All items are loaded including failed ones on retry.
+----------------+----------------------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------------------------+--------------+
|
| 172.23.104.203 | [u'kv'] | Cluster node |
|
| 172.23.104.202 | [u'index', u'kv', u'n1ql'] | Cluster node |
|
| 172.23.104.208 | None | <--- IN --- |
|
| 172.23.104.211 | None | <--- IN --- |
|
+----------------+----------------------------+--------------+
|
4. Repeat step 3. Rebalance in 2 more node and load 50k more docs with durability=majority.
Again many DurabilityImpossibleExceptions are seen.
+----------------+----------------------------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------------------------+--------------+
|
| 172.23.104.211 | [u'kv'] | Cluster node |
|
| 172.23.104.203 | [u'kv'] | Cluster node |
|
| 172.23.104.208 | [u'kv'] | Cluster node |
|
| 172.23.104.202 | [u'index', u'kv', u'n1ql'] | Cluster node |
|
| 172.23.104.216 | None | <--- IN --- |
|
| 172.23.104.217 | None | <--- IN --- |
|
+----------------+----------------------------+--------------+
|
5. Here are the documents which failed during rebalance with DurabilityAmbiguousException and after rebalance is finished these documents are retried but failed again with the same exception:
Docs failed after expected retry for 'create' (143): [u'test_docs-117754', u'test_docs-122707', u'test_docs-144857', u'test_docs-116029', u'test_docs-117355', u'test_docs-140775', u'test_docs-149228', u'test_docs-121211', u'test_docs-110961', u'test_docs-110289', u'test_docs-124290', u'test_docs-145664', u'test_docs-109175', u'test_docs-129342', u'test_docs-143481', u'test_docs-148262', u'test_docs-129696', u'test_docs-135299', u'test_docs-122306', u'test_docs-141903', u'test_docs-109607', u'test_docs-140007', u'test_docs-111421', u'test_docs-143511', u'test_docs-108871', u'test_docs-129074', u'test_docs-143110', u'test_docs-149850', u'test_docs-130218', u'test_docs-104391', u'test_docs-129743', u'test_docs-146590', u'test_docs-127565', u'test_docs-122030', u'test_docs-143080', u'test_docs-102643', u'test_docs-146191', u'test_docs-109797', u'test_docs-128935', u'test_docs-115398', u'test_docs-111353', u'test_docs-146337', u'test_docs-143226', u'test_docs-148510', u'test_docs-145086', u'test_docs-116922', u'test_docs-140197', u'test_docs-127653', u'test_docs-148480', u'test_docs-104573', u'test_docs-124300', u'test_docs-123971', u'test_docs-130388', u'test_docs-123698', u'test_docs-134343', u'test_docs-112172', u'test_docs-142814', u'test_docs-117063', u'test_docs-145116', u'test_docs-148626', u'test_docs-136128', u'test_docs-148227', u'test_docs-121563', u'test_docs-115208', u'test_docs-146445', u'test_docs-147895', u'test_docs-110319', u'test_docs-124036', u'test_docs-127786', u'test_docs-104201', u'test_docs-124437', u'test_docs-105539', u'test_docs-144812', u'test_docs-117711', u'test_docs-140331', u'test_docs-121655', u'test_docs-145621', u'test_docs-140730', u'test_docs-121254', u'test_docs-145220', u'test_docs-108798', u'test_docs-117681', u'test_docs-122742', u'test_docs-141946', u'test_docs-104879', u'test_docs-140443', u'test_docs-111065', u'test_docs-145858', u'test_docs-141893', u'test_docs-143554', u'test_docs-142984', u'test_docs-111464', u'test_docs-139979', u'test_docs-129031', u'test_docs-136866', u'test_docs-129430', u'test_docs-140042', u'test_docs-121381', u'test_docs-135309', u'test_docs-122075', u'test_docs-128970', u'test_docs-122474', u'test_docs-129706', u'test_docs-137162', u'test_docs-147905', u'test_docs-115870', u'test_docs-103609', u'test_docs-112244', u'test_docs-143263', u'test_docs-148555', u'test_docs-146773', u'test_docs-143662', u'test_docs-116967', u'test_docs-111316', u'test_docs-146372', u'test_docs-148154', u'test_docs-111717', u'test_docs-133977', u'test_docs-124345', u'test_docs-127616', u'test_docs-124744', u'test_docs-123934', u'test_docs-121526', u'test_docs-123708', u'test_docs-117427', u'test_docs-121127', u'test_docs-111286', u'test_docs-117026', u'test_docs-111687', u'test_docs-145552', u'test_docs-146400', u'test_docs-112137', u'test_docs-144982', u'test_docs-146001', u'test_docs-145153', u'test_docs-124472', u'test_docs-122296', u'test_docs-122697', u'test_docs-134431', u'test_docs-108608', u'test_docs-103799', u'test_docs-140949', u'test_docs-103835']