Description
call packages\sdkdclient\bin\brun.bat -A S3Creds_tmp -I cluster_conf
ig.ini -I sdkd.args --variants HYBRID -d all:trace
[0.30 INFO] (BRun run:436) Initializing S3
[1.12 INFO] (BRun run:441) Initializing history database
[1.75 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools
============================================================
Running ReAdd2-HYBRID: 2014-08-26 22:31:28
workload=HybridWorkloadGroup
failover/next_action=FO_READD_REBALANCE
failover/count=2
failover/next_delay=45
testcase=FailoverScenario
Logging to c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1174\ReAdd2-HYBRID\08-26-14\0198
88\0dcd5a5bc61e625f06a757ab9dfa7db5
To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
\ --workload HybridWorkloadGroup --failover-next_action
\ FO_READD_REBALANCE --failover-count 2 --failover-next_delay 45
\ --testcase FailoverScenario -C share\rexec --rexec_path
\ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
\ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
\ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
--cluster_ssh-username=root --cluster_ssh-password=couchbase
.....
[123.71 INFO] (Scenario run:86) Starting CHANGE phase
[123.71 INFO] (CBCluster failoverNodes:588) Failing over <URI:10.3.121.135:8091,ns_1@10.3.121.135>
[123.72 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/failOver
[124.10 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[124.13 INFO] (CBCluster failoverNodes:588) Failing over <URI:10.3.121.136:8091,ns_1@10.3.121.136>
[124.14 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/failOver
[124.40 INFO] (FailoverAction change:104) Nodes failed over. Requested sleep for 45 seconds
[125.61 INFO] (SDKD log:137)
[125.63 INFO] (SDKD log:137) Unhandled Exception: Couchbase.ServerUnavailableException: Exception of type 'Couchbase.ServerUnavail
ableException' was thrown.
[169.41 INFO] (FailoverAction doNextAction:68) Readding nodes after failover
[169.41 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/reAddNode
[169.41 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/reAddNode
[169.42 INFO] (FailoverAction doNextAction:85) Rebalancing nodes after failover
[169.42 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/nodes
[169.43 TRACE] (CouchbaseAdmin getResponseJson:167) POST http://10.3.121.134:8091/controller/rebalance
[169.45 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
[169.45 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0%
[170.46 TRACE] (CouchbaseAdmin getResponseJson:167) GET
....
[196.67 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
[196.68 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 37%
[197.59 INFO] (SDKD log:137) [Sdkd.ViewQuery|Error] ... System.NullReferenceException Object reference not set to an instance of a
n object. at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Views\ViewComman
d.cs:line 268
[197.61 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[197.61 INFO] (SDKD log:137) at Couchbase.Configuration.ConfigContextBase.GetServer() in c:\jenkins\workspace\.net-sdkdclient-n
g2-andrei\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 210
at Couchbase.Core.Buckets.CouchbaseBucket.HandleIOError[T](IOperation`1 operation, IServer server) in c:\jenkins\workspace\.net
-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 194
at Couchbase.Core.Buckets.CouchbaseBucket.CanRetryOperation[T](IOperationResult`1 operationResult, IOperation`1 operation, ISer
ver server) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:
line 179
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) in c:\jenkins\workspace\.net-sdkdclient-ng2-
andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 104
at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\couchbase-net-c
lient\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 888
at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net
\src\Sdkd\Commands\GetCommand.cs:line 18
at Sdkd.Commands.KVCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\Sdkd\Commands\KVCommand.cs:lin
e 25
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2-andr
ei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 295
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2-andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs
:line 175
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boo
lean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean pre
serveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
[197.64 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_27]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_27]
at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_27]
[197.66 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.73
[197.66 TRACE] (Handle sendMessageAsync:185) > {"Command":"CLOSEHANDLE","Handle":103,"ReqID":73,"CommandData":{}}
[197.66 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_27]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_27]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_27]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_27]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_27]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]
[197.68 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
[197.83 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 39%
[198.80 INFO] (BRun uploadFile:141) Uploaded RAW => http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1174/Re
Add2-HYBRID/08-26-14/019888/0dcd5a5bc61e625f06a757ab9dfa7db5.zip
[198.83 TRACE] (CouchbaseAdmin getResponseJson:167) GET http://10.3.121.134:8091/pools/default/rebalanceProgress
Attachments
Issue Links
- is duplicated by
-
NCBC-598 ServerUnavailableException after rebalance (in CouchbaseBucket.Get)
- Closed