This problem occurred during 2 kv node eject and 2 new nodes added. Eventing is processing mutations and I can see 2 bucketop failing with LCB_PROTOCOL_ERROR
1.
2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] [lcb,retryq L:163 I:706008921] Failing command (seq=6602) from retry queue: LCB_PROTOCOL_ERROR (0x16)
2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Bucket: LCB_STORE breaking out
2019-11-28T10:48:21.252-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5 ^Error at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)
2.
2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] [lcb,retryq L:163 I:1681454432] Failing command (seq=6774) from retry queue: LCB_PROTOCOL_ERROR (0x16)
2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Bucket: LCB_STORE breaking out
2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5 ^Error at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)
Due to this we could not perform the bucketop successfully and hence 2 docs missing. The LCB_PROTOCOL_ERROR seems related to some transient network problems, but eventing cannot retry on this error since there are non-retryable errors based on this error code.
Faiz Veettil Could you please check if it is timers related? If yes, it may be a duplicate of
MB-36468.