Uploaded image for project: 'Couchbase .NET client library'
  1. Couchbase .NET client library
  2. NCBC-606

Couchbase.ServerUnavailableException at Couchbase.Configuration.ConfigContextBase.GetServer() when some nodes failed over

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 2.0 Beta
    • 2.0 DP3
    • library
    • None

    Description

      call packages\sdkdclient\bin\brun.bat -A S3Creds_tmp -I cluster_conf
      ig.ini -I sdkd.args --variants HYBRID -d all:trace
      [0.30 INFO] (BRun run:436) Initializing S3
      [1.12 INFO] (BRun run:441) Initializing history database
      [1.75 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools

      ============================================================
      Running ReAdd2-HYBRID: 2014-08-26 22:31:28
      workload=HybridWorkloadGroup
      failover/next_action=FO_READD_REBALANCE
      failover/count=2
      failover/next_delay=45
      testcase=FailoverScenario
      Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1174\ReAdd2-HYBRID\08-26-14\0198
      88\0dcd5a5bc61e625f06a757ab9dfa7db5

      To re-run the test, copy/paste the following into the shell.
      You may also copy/paste (except the first line) into an argfile
      8<----------------------------------------
      ./stester \
      \ --workload HybridWorkloadGroup --failover-next_action
      \ FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
      \ --testcase FailoverScenario -C share\rexec --rexec_path
      \ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
      \ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
      \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
      --cluster_ssh-username=root --cluster_ssh-password=couchbase

      .....

      [123.71 INFO] (Scenario run:86) Starting CHANGE phase
      [123.71 INFO] (CBCluster failoverNodes:588) Failing over <URI:10.3.121.135:8091,ns_1@10.3.121.135>
      [123.72 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/failOver
      [124.10 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
      [124.13 INFO] (CBCluster failoverNodes:588) Failing over <URI:10.3.121.136:8091,ns_1@10.3.121.136>
      [124.14 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/failOver
      [124.40 INFO] (FailoverAction change:104) Nodes failed over. Requested sleep for 45 seconds
      [125.61 INFO] (SDKD log:137)
      [125.63 INFO] (SDKD log:137) Unhandled Exception: Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavail
      ableException' was thrown.
      [169.41 INFO] (FailoverAction doNextAction:68) Readding nodes after failover
      [169.41 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/reAddNode
      [169.41 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/reAddNode
      [169.42 INFO] (FailoverAction doNextAction:85) Rebalancing nodes after failover
      [169.42 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/nodes
      [169.43 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/rebalance
      [169.45 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
      [169.45 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
      [170.46 TRACE] (CouchbaseAdmin getResponseJson:167) GET
      ....
      [196.67 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
      [196.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 37%
      [197.59 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of a
      n object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewComman
      d.cs:line 268
      [197.61 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
      [197.61 INFO] (SDKD log:137) at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-n
      g2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
      at Couchbase.Core.Buckets.CouchbaseBucket.HandleIOError[T](IOperation`1 operation, IServer server) in c:\jenkins\workspace\.net
      -sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 194
      at Couchbase.Core.Buckets.CouchbaseBucket.CanRetryOperation[T](IOperationResult`1 operationResult, IOperation`1 operation, ISer
      ver server) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:
      line 179
      at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-
      andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 104
      at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-c
      lient\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 888
      at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net
      \src\Sdkd\Commands\GetCommand.cs:line 18
      at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Commands\KVCommand.cs:lin
      e 25
      at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2-andr
      ei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 295
      at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs
      :line 175
      at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
      at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boo
      lean preserveSyncCtx)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean pre
      serveSyncCtx)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
      at System.Threading.ThreadHelper.ThreadStart()
      [197.64 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
      com.couchbase.sdkdclient.context.HarnessException: Driver crashed
      at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
      at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
      at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
      at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
      at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
      [197.66 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.73
      [197.66 TRACE] (Handle sendMessageAsync:185) > {"Command":"CLOSEHANDLE","Handle":103,"ReqID":73,"CommandData":{}}
      [197.66 WARN] (STester close:170) While stopping driver
      java.net.SocketException: Connection reset by peer: socket write error
      at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
      at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
      at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
      at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
      at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
      at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
      at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
      at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
      at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
      at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
      at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]
      [197.68 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
      [197.83 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 39%
      [198.80 INFO] (BRun uploadFile:141) Uploaded RAW => http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1174/Re
      Add2-HYBRID/08-26-14/019888/0dcd5a5bc61e625f06a757ab9dfa7db5.zip
      [198.83 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress

      Attachments

        1. NCBC-606.zip
          6.02 MB
        2. test.logs
          125 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jmorris Jeff Morris
              andreibaranouski Andrei Baranouski
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty