Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-649

infinite loop 'Trying again' with 'No connections currently available' for ReAdd2 test when nodes returned to cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Test Blocker
    • 2.0 Beta 2
    • 2.0 Beta
    • library
    • None

    Description

      run with commit https://github.com/couchbase/couchbase-net-client/commit/24714876f76207a5fc7d72e88bd1f9c57909fcdb

      http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/102/console

      Running ReAdd2-HYBRID: 2014-09-18 23:50:57
      workload=HybridWorkloadGroup
      failover/next_action=FO_READD_REBALANCE
      failover/count=2
      failover/next_delay=45
      testcase=FailoverScenario
      Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\ReAdd2-HYBRID\09-18-14\024657\8c847f52905fd0bf669803f16c2dc716

      To re-run the test, copy/paste the following into the shell.
      You may also copy/paste (except the first line) into an argfile
      8<----------------------------------------
      ./stester \
      --workload HybridWorkloadGroup --failover-next_action
      \
      FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
      \
      --testcase FailoverScenario -C share\rexec --rexec_path
      \
      C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
      \
      --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
      \
      --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
      \
      --cluster_ssh-username=root --cluster_ssh-password=couchbase

      2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
      2014-09-18 23:56:46,364 [2] DEBUG Couchbase.CouchbaseCluster - Finalizing CouchbaseCluster
      2014-09-18 23:56:46,614 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
      at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
      at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
      2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 1 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
      2014-09-18 23:56:46,614 [42] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#878 on thread 42
      2014-09-18 23:56:46,614 [42] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 609 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 - rev878
      2014-09-18 23:56:46,614 [42] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP4463REP4463REP4463REP4463REP4463 using server 10.3.121.134:11210
      2014-09-18 23:56:46,615 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b | 10.3.121.134:11210 | [0, 2] - b20c68a7-f922-486e-af16-f880d88bfff6 - Disposed: False
      2014-09-18 23:56:46,615 [42] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP4463REP4463REP4463REP4463REP4463 using 609 on 10.3.121.134:11210
      2014-09-18 23:56:46,616 [42] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 2059d0b9-eb1e-4163-b58a-17ae44f4ae8b on 10.3.121.134:11210 - b20c68a7-f922-486e-af16-f880d88bfff6
      2014-09-18 23:56:46,619 [42] INFO Couchbase.Core.Buckets.CouchbaseBucket - New config found 1736
      2014-09-18 23:56:46,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:56:46,880 [42] ERROR Couchbase.Core.Buckets.CouchbaseBucket - Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavailableException' was thrown.
      at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
      at Couchbase.Core.Buckets.CouchbaseBucket.CheckForConfigUpdates[T](IOperationResult`1 operationResult, IOperation operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 241
      2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation retry 2 for key SimpleKeyREP4463REP4463REP4463REP4463REP4463. Reason: VBucketBelongsToAnotherServer
      2014-09-18 23:56:46,880 [42] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP4463REP4463REP4463REP4463REP4463 failed after 3 retries. Reason: VBucketBelongsToAnotherServer
      2014-09-18 23:56:49,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:56:51,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:56:54,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:56:56,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:56:59,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:57:01,646 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-18 23:57:04,146 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False

      ..................

      2014-09-19 01:09:51,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-19 01:09:54,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-19 01:09:56,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-19 01:09:59,304 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False
      2014-09-19 01:10:01,804 [45] DEBUG Couchbase.IO.ConnectionPool`1 - No connections currently available on 10.3.121.136:11210 - ae3a8fe3-d2b0-4a20-83bd-512e9c870573. Trying again. - Disposed: False

      test stuck, not all items loaded
      run was via jenkins, so I'm not able to get stacktraces

      Attachments

        1. NCBC-649_1.png
          NCBC-649_1.png
          155 kB
        2. NCBC-649_2.png
          NCBC-649_2.png
          101 kB
        3. NCBC-649_3.png
          NCBC-649_3.png
          119 kB
        4. NCBC-649_4.png
          NCBC-649_4.png
          159 kB
        5. NCBC-649_gerrit.7z
          35.85 MB
        6. NCBC-649.7z
          38.98 MB
        7. NCBC-649.png
          NCBC-649.png
          75 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jmorris Jeff Morris
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty