Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-37097

Mutation is failing with COUCHBASE_PROTOCOL_ERROR 22

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not a Bug
    • 6.5.0
    • 6.5.0
    • eventing
    • Untriaged
    • Unknown

    Description

      Build: 6.5.0-4897

       ./testrunner -i /tmp/testexec.14899.ini -p get-cbcollect-info=True,GROUP=bucket_op -t eventing.eventing_rebalance.EventingRebalance.test_eventing_rebalance_with_multiple_kv_nodes,doc-per-day=5,dataset=default,nodes_init=5,services_init=kv-kv-kv-eventing-index:n1ql,groups=simple,reset_services=True,GROUP=bucket_op
       
      Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 10078 Expected : 10080  dcp_backlog : 0  TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.104.202': {u'22': 2}} 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Faiz Veettil Could you please check if it is timers related? If yes, it may be a duplicate of MB-36468.

          jeelan.poola Jeelan Poola added a comment - Faiz Veettil Could you please check if it is timers related? If yes, it may be a duplicate of MB-36468 .

          This problem occurred during 2 kv node eject and 2 new nodes added. Eventing is processing mutations and I can see 2 bucketop failing with LCB_PROTOCOL_ERROR

          1.

          2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] [lcb,retryq L:163 I:706008921] Failing command (seq=6602) from retry queue: LCB_PROTOCOL_ERROR (0x16)

          2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Bucket: LCB_STORE breaking out

          2019-11-28T10:48:21.252-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5                       ^Error    at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)

           

          2.

          2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] [lcb,retryq L:163 I:1681454432] Failing command (seq=6774) from retry queue: LCB_PROTOCOL_ERROR (0x16)

          2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Bucket: LCB_STORE breaking out

          2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5                       ^Error    at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)

          Due to this we could not perform the bucketop successfully and hence 2 docs missing. The LCB_PROTOCOL_ERROR seems related to some transient network problems, but eventing cannot retry on this error since there are non-retryable errors based on this error code.

           

          faiz.veettil Faiz Veettil (Inactive) added a comment - This problem occurred during 2 kv node eject and 2 new nodes added. Eventing is processing mutations and I can see 2 bucketop failing with LCB_PROTOCOL_ERROR 1. 2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] [lcb,retryq L:163 I:706008921] Failing command (seq=6602) from retry queue: LCB_PROTOCOL_ERROR (0x16) 2019-11-28T10:48:21.251-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Bucket: LCB_STORE breaking out 2019-11-28T10:48:21.252-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_0:/tmp/127.0.0.1:8091_0_2465623617.sock:125357] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5                       ^Error    at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24)   2. 2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] [lcb,retryq L:163 I:1681454432] Failing command (seq=6774) from retry queue: LCB_PROTOCOL_ERROR (0x16) 2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Bucket: LCB_STORE breaking out 2019-11-28T10:48:21.299-08:00 [Info] eventing-consumer [worker_Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes_1:/tmp/127.0.0.1:8091_1_2465623617.sock:125366] Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5                       ^Error    at OnUpdate (Function_104570868_test_eventing_rebalance_with_multiple_kv_nodes.js:5:24) Due to this we could not perform the bucketop successfully and hence 2 docs missing. The LCB_PROTOCOL_ERROR seems related to some transient network problems, but eventing cannot retry on this error since there are non-retryable errors based on this error code.  

          People

            faiz.veettil Faiz Veettil (Inactive)
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty