Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.5.2, 6.6.5, 7.0.0, 7.1.0, 7.1.1
-
7.1.0-2506
-
Untriaged
-
-
1
-
No
-
KV March-22, KV May 22
Description
1. Create a 3 node cluster and a default bucket.
2. Load 100k items in a magma bucket with durability=MAJORITY_AND_PERSIST_TO_ACTIVE
3. Start a durable upsert load in async.
4. Rebalance in 1 node. while upserts are running. Retry the failed documents due to DurabilityAmbiguousException.
5. Start a durable upsert load in async.
6. Rebalance out 1 node. while upserts are running. Retry the failed documents due to DurabilityAmbiguousException.
7. Retry failed.
>>>
|
>>> r = cb_coll.upsert("test_docs-4991", "") |
Traceback (most recent call last):
|
File "<stdin>", line 1, in <module> |
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/collection.py", line 293, in wrapped |
return func(self, *args, **kwargs) |
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/result.py", line 682, in mutated |
result = func(*args, **kwargs)
|
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/collection.py", line 878, in upsert |
return ResultPrecursor(CoreClient.upsert( |
couchbase.exceptions.CouchbaseException: <Key='test_docs-4991', RC=0x136[LCB_ERR_DURABLE_WRITE_IN_PROGRESS (310)], Operational Error, Results=1, C Source=(src/multiresult.c,332), Context={'status_code': 4, 'opaque': 3, 'cas': 0, 'key': 'test_docs-4991', 'bucket': 'default', 'collection': '_default', 'scope': '_default', 'context': '', 'ref': '', 'endpoint': '172.23.121.116:11210', 'type': 'KVErrorContext'}, Tracing Output={"test_docs-4991": {"debug_info": {"FILE": "src/callbacks.c", "FUNC": "dur_chain2", "LINE": 751}}}> |
>>>
|
>>>
|
>>> r = cb_coll.get("test_docs-4991") |
Traceback (most recent call last):
|
File "<stdin>", line 1, in <module> |
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/collection.py", line 293, in wrapped |
return func(self, *args, **kwargs) |
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/result.py", line 544, in wrapped |
x, options = func(*args, **kwargs)
|
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/collection.py", line 521, in get |
return self._get_generic(key, kwargs, options) |
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase/collection.py", line 484, in _get_generic |
x = CoreClient.get(self.bucket, key, **opts)
|
File "/Users/ritesh.agarwal/Library/Python/3.8/lib/python/site-packages/couchbase_core/client.py", line 409, in get |
return super(Client, self).get(*args, **kwargs) |
couchbase.exceptions.CouchbaseException: <Key='test_docs-4991', RC=0x137[LCB_ERR_DURABLE_WRITE_RE_COMMIT_IN_PROGRESS (311)], Operational Error, Results=1, C Source=(src/multiresult.c,332), Context={'status_code': 164, 'opaque': 7, 'cas': 0, 'key': 'test_docs-4991', 'bucket': 'default', 'collection': '_default', 'scope': '_default', 'context': '', 'ref': '', 'endpoint': '172.23.121.116:11210', 'type': 'KVErrorContext'}, Tracing Output={"test_docs-4991": {"debug_info": {"FILE": "src/callbacks.c", "FUNC": "value_callback", "LINE": 856}}}> |
>>>
|
8. The document is stuck like this forever.
Note: Not really sure if this is a regression as this looks like the test is passing occasionally. Trying on 7.0 GA build.
guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/magma_temp_job4.ini -p GROUP=P0;durability,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,bucket_storage=magma,rerun=False,enable_dp=false,autoCompactionDefined=true,infra_log_level=debug,log_level=debug,bucket_storage=magma,upgrade_version=7.1.0-2506 -t rebalance_new.rebalance_in_out.RebalanceInOutTests.test_incremental_rebalance_out_in_with_mutation,upgrade_version=7.1.0-2506,rerun=False,enable_dp=false,GROUP=P0;durability,get-cbcollect-info=False,replicas=1,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,bucket_storage=magma,log_level=debug,nodes_init=3,autoCompactionDefined=true,infra_log_level=debug'
|
Failing on couchstore as well:
http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.1.0-2284/jenkins_logs/test_suite_executor-TAF/169859/consoleText.txt
http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.1.0-1601/jenkins_logs/test_suite_executor-TAF/152135/consoleText.txt
Attachments
Issue Links
For Gerrit Dashboard: MB-51606 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
172919,4 | MB-51606: Notify DM for persistence of dead vBuckets | master | kv_engine | Status: ABANDONED | +2 | +1 |
172940,1 | MB-51606: Split setupConsumerAndPassiveStream fn into components | master | kv_engine | Status: ABANDONED | 0 | +1 |
172988,2 | MB-51606: Don't throw if PDM of dead vBucket tries to ack | master | kv_engine | Status: ABANDONED | -1 | -1 |
174463,2 | MB-51606: Split setupConsumerAndPassiveStream fn into components | neo | kv_engine | Status: MERGED | +2 | +1 |
174464,2 | MB-51606: Notify DM for persistence of dead vBuckets | neo | kv_engine | Status: MERGED | +2 | +1 |
174465,6 | MB-51606: Don't throw if PDM of dead vBucket tries to ack | neo | kv_engine | Status: MERGED | +2 | +1 |
174506,5 | MB-51606: Update seqno ack to send outside state lock | neo | kv_engine | Status: MERGED | +2 | +1 |
174538,3 | MB-51606: Pass VB State lock to DM::notifyLocalPersistence() | neo | kv_engine | Status: MERGED | +2 | +1 |
174539,4 | MB-51606: Pass VB State lock to PDM::notifySnapshotEndReceived() | neo | kv_engine | Status: MERGED | +2 | +1 |
174540,4 | MB-51606: Pass VB State lock to VBucket::sendSeqnoAck() | neo | kv_engine | Status: MERGED | +2 | +1 |
174799,5 | MB-51606: Fixup ADM state on transition to PDM | neo | kv_engine | Status: MERGED | +2 | +1 |
175268,3 | Revert initial fix for MB-51606 | neo | kv_engine | Status: MERGED | +2 | +1 |
175749,3 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
176098,2 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
176177,2 | MB-51606: Split setupConsumerAndPassiveStream fn into components | neo | kv_engine | Status: MERGED | +2 | +1 |
176178,2 | MB-51606: Use DM interface functions instead of casting to child | neo | kv_engine | Status: MERGED | +2 | +1 |
176179,2 | MB-51606: Add VBucket& getter to DM interface | neo | kv_engine | Status: ABANDONED | -1 | +1 |
176180,5 | MB-51606: Change ADM::getTrackedWrites() to getTrackedKeys() | neo | kv_engine | Status: MERGED | +2 | +1 |
176184,5 | MB-51606: Create DM from other base class DM | neo | kv_engine | Status: MERGED | +2 | +1 |
176192,10 | MB-51606: Add DeadDurabilityMonitor | neo | kv_engine | Status: MERGED | +2 | +1 |
176193,16 | MB-51606: Notify PDM of last consistent point on transition | neo | kv_engine | Status: MERGED | +2 | +1 |
176220,9 | MB-51606: Move queueing of snapshot end into new PDM::State fn | neo | kv_engine | Status: MERGED | +2 | +1 |
176362,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
176456,1 | Merge commit 'neo' into master | master | kv_engine | Status: ABANDONED | 0 | 0 |
176459,1 | Merge commit 'neo' into master | master | kv_engine | Status: MERGED | +2 | +1 |
176518,1 | Merge commit 'neo' into master | master | kv_engine | Status: ABANDONED | 0 | 0 |
176522,1 | Merge commit '7230ffe6' into master | master | kv_engine | Status: MERGED | +2 | +1 |
176536,1 | Merge branch 'neo' | master | kv_engine | Status: ABANDONED | 0 | -1 |
176550,3 | MB-51606: Don't set lastSeqno to 1000 in VBucketTest | neo | kv_engine | Status: MERGED | +2 | +1 |
176551,2 | MB-51606: Don't set receivedSnapshotEnd in PDM ctor if seqno is 0 | neo | kv_engine | Status: MERGED | +2 | +1 |
176552,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |
176602,1 | Merge branch 'neo' | master | kv_engine | Status: MERGED | +2 | +1 |