Uploaded image for project: 'Java DCP Client'
  1. Java DCP Client
  2. JDCP-15

Client.connect().await() hangs intermittently

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 0.5.0
    • 0.2.0, 0.3.0, 0.4.0
    • three-node CB cluster (4.5.0-enterprise)

    Description

      CouchbaseConnectorStressTest.testStreamLifecycleSingleThread performs the following in a loop:

                      Client client = Client.configure()
                              .eventLoopGroup(CBTestEnvironmentProvider.getEnvironment().ioPool())
                              .bucket(bucket)
                              .password(cbPassword)
                              .hostnames(cbNodes)
                              .connectionNameGenerator(() -> threadName)
                              .controlParam(DcpControl.Names.CONNECTION_BUFFER_SIZE, 16384)
                              .bufferAckWatermark(80)
                              .build();
       
                      client.controlEventHandler(ReferenceCountUtil::release);
                      client.dataEventHandler(event -> {
                          try {
                              client.acknowledgeBuffer(event);
                          } finally {
                              ReferenceCountUtil.release(event);
                          }
                      });
       
                      timedAwait("client.connect()", client.connect());
                      timedAwait("client.disconnect()", client.disconnect());
      

      This intermittently hangs (times out after 90s) on the client.connect()- observed on the initial or later iterations.
      Example thread dump when await() times out attached.

      See also Jenkins log here: http://analytics.jenkins.couchbase.com/job/couchsterix-test/com.couchbase.analytics$cbas-ingestion-test/912/consoleFull. The same failure occurs in both the single and multi-threadded tests on that Jenkins run.

      Observed in both 0.2.0 and 0.3.0-SNAPSHOT (f259c97)

      Repro instructions to follow.

      Attachments

        1. cc-dcp-debug.log
          22.47 MB
        2. cc-dcp-debug-1240.log
          13.64 MB
        3. cc-dcp-debug-1240-part.log
          4.71 MB
        4. cc-dcp-debug-one-off-1343-part.log
          8.83 MB
        5. log-jira.txt
          342 kB
        6. thread_dump_912.rtf
          52 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              avsej Sergey Avseyev
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty