Uploaded image for project: 'Java Couchbase JVM Core'
  1. Java Couchbase JVM Core
  2. JVMCBC-708

Client not dispatching doc operation requests to active vbucket after memcached got restarted and getting 'RequestTimeoutException' for undispatched docs

    XMLWordPrintable

Details

    Description

      Server build: 6.5.0-3821

      Client build:

      core-io-2.0.0-alpha.5.jar
      java-client-3.0.0-alpha.5.jar

      Scenario:

      1. 4 node cluster, Couchbase bucket (replicas=2)
      2. Start loading documents targeting the active vbuckets persent on node-1 (10.112.191.101)
      3. Kill Memcached when doc_loading is running in background

      Observation:

      Memcached on node-1 recovers successfully. But new doc_create requests are not dispatched from clients's end (Verified using doc_req.pcapng).

      Seeing 'com.couchbase.client.core.error.RequestTimeoutException' for un-dispatched requests from client end.

      Expected behavior:

      Doc create request should be resumed once the memcached restarted successfully in cb-server.

      Server logs:
      https://s3.amazonaws.com/cb-engineering/jira/request_dispatched_post_active_mc_restart/collectinfo-2019-07-22T072137-ns_1%4010.112.191.101.zip
      https://s3.amazonaws.com/cb-engineering/jira/request_dispatched_post_active_mc_restart/collectinfo-2019-07-22T072137-ns_1%4010.112.191.102.zip
      https://s3.amazonaws.com/cb-engineering/jira/request_dispatched_post_active_mc_restart/collectinfo-2019-07-22T072137-ns_1%4010.112.191.103.zip
      https://s3.amazonaws.com/cb-engineering/jira/request_dispatched_post_active_mc_restart/collectinfo-2019-07-22T072137-ns_1%4010.112.191.104.zip
       
      TAF testcase:

      crash_test.crash_process.CrashTest.test_crash_process,nodes_init=4,replicas=2,num_items=10000,process=memcached,service=memcached,sig_type=sigkill,target_node=active,durability=MAJORITY

      Attachments

        1. doc_req.pcapng
          8.52 MB
        2. JavaSDK_alpha-6_Snapshot.log
          116 kB
        3. JavaSDK.log
          9.63 MB
        4. test.log
          480 kB

        Activity

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty