Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Not a Bug
-
6.5.0
-
Untriaged
-
Unknown
Description
Build: 6.5.0-4897
./testrunner -i /tmp/testexec.14899.ini -p get-cbcollect-info=True,GROUP=bucket_op -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,GROUP=bucket_op
|
|
|
Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 10078 Expected : 10080 dcp_backlog : 0 TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.104.202': {u'22': 2}}
|
Attachments
Activity
This problem occurred during 2 kv node eject and 2 new nodes added. Eventing is processing mutations and I can see 2 bucketop failing with LCB_PROTOCOL_ERROR
1.
2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] [lcb,retryq L:163 I:706008921] Failing command (seq=6602) from retry queue: LCB_PROTOCOL_ERROR (0x16)
2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Bucket: LCB_STORE breaking out
2019-11-28T10:48:21.252-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5 ^Error at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)
2.
2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] [lcb,retryq L:163 I:1681454432] Failing command (seq=6774) from retry queue: LCB_PROTOCOL_ERROR (0x16)
2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Bucket: LCB_STORE breaking out
2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5 ^Error at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)
Due to this we could not perform the bucketop successfully and hence 2 docs missing. The LCB_PROTOCOL_ERROR seems related to some transient network problems, but eventing cannot retry on this error since there are non-retryable errors based on this error code.
Faiz Veettil Could you please check if it is timers related? If yes, it may be a duplicate of
MB-36468.