Description
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/23/consoleFull
Running SvcRestart-HYBRID: 2014-09-11 23:57:30
service/restore_delay=5
workload=HybridWorkloadGroup
service/name=SYSV_SERVICE
service/action=kill
service/count=3
testcase=ServiceFailureScenario
Logging to c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\SvcRestart-HYBRID\09-11-14\025050\880a91bc80225536124352f8115cbc71
To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
--service-restore_delay 5 --workload HybridWorkloadGroup
\ --service-name SYSV_SERVICE --service-action kill --service-count
\ 3 --testcase ServiceFailureScenario -C share\rexec --rexec_path
\ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
\ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
\ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
\ --cluster_ssh-username=root --cluster_ssh-password=couchbase
.....
[2157.71 INFO] (Scenario run:83) [39mRAMP phase started. Waiting for 30 seconds[0;39m
[2157.71 INFO] (SDKD log:137) [39m[Sdkd.Handle|Warn] Got command CB_VIEW_QUERY [0;39m
[2157.71 INFO] (SDKD log:137) [39m[Sdkd.Handle|Warn] Command data: {[0;39m
[2157.71 INFO] (SDKD log:137) [39m "DSType": "DSTYPE_SEEDED",[0;39m
[2157.71 INFO] (SDKD log:137) [39m "DS": {},[0;39m
[2157.71 INFO] (SDKD log:137) [39m "ViewParameters":
,[0;39m
[2157.71 INFO] (SDKD log:137) [39m "DesignName": "test_design",[0;39m
[2157.71 INFO] (SDKD log:137) [39m "Options":
,[0;39m
[2157.71 INFO] (SDKD log:137) [39m "ViewName": "test_view"[0;39m
[2157.71 INFO] (SDKD log:137) [39m} [0;39m
[2187.71 INFO] (Scenario run:86) [39mStarting CHANGE phase[0;39m
[2187.72 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server stop on 10.3.121.135[0;39m
[2190.35 INFO] (SDKD log:137) [39m[Sdkd.Result|Warn] Unrecognized return code 409 [0;39m
[2191.77 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@52d654[0;39m
[2192.77 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server stop on 10.3.121.136[0;39m
[2196.16 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@38c8c5[0;39m
[2197.19 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server stop on 10.3.3.206[0;39m
[2201.34 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@f522e8[0;39m
[2202.34 INFO] (Scenario sleepAndDo:109) [39mSleeping 5 seconds[0;39m
[2207.38 INFO] (Scenario executeChange:134) [39mTurning on services for nodes[0;39m
[2207.38 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server start on 10.3.121.135[0;39m
[2208.94 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@19c873d[0;39m
[2209.96 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server start on 10.3.121.136[0;39m
[2211.47 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@12a2259[0;39m
[2212.47 DEBUG] (SSHCommand execute:75) [34mRunning /etc/init.d/couchbase-server start on 10.3.3.206[0;39m
[2214.05 DEBUG] (SSHCommand close:147) [34mClosing channel com.jcraft.jsch.ChannelExec@fe0ce9[0;39m
[2215.05 INFO] (Scenario run:97) [39mCHANGE phase done[0;39m
[2215.05 INFO] (Scenario run:103) [39mStarting REBOUND for 90 seconds[0;39m
[2223.95 INFO] (SDKD log:137) [39m[0;39m
[2223.95 INFO] (SDKD log:137) [39m[0;39m
[2224.03 INFO] (SDKD log:137) [39mUnhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: [0;39m
[2224.04 INFO] (SDKD log:137) [39mUnhandled Exception: System.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (LineGobbler doFilter:115) [39m+++ Following exception has internal ID: 1[0;39m
[2224.10 INFO] (SDKD log:137) [39m at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 0
at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (LineGobbler doFilter:115) [39m+++ Following exception has internal ID: 3[0;39m
[2224.10 INFO] (SDKD log:137) [39m at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1141
at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (SDKD log:137) [39m+++ Received exception of ID 2[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (SDKD log:137) [39m+++ Received exception of ID 2[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (SDKD log:137) [39m+++ Received exception of ID 2[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (SDKD log:137) [39m+++ Received exception of ID 2[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2224.10 INFO] (SDKD log:137) [39m+++ Received exception of ID 2[0;39m
[2224.10 INFO] (SDKD log:137) [39mSystem.OperationCanceledException: The operation was canceled.[0;39m
[2235.04 INFO] (SDKD log:137) [39m+++ Flushing remaining exception trace buffer[0;39m
[2235.04 ERROR] (BRun maybePropagate:243) [1;31mSuppressing driver-related error[0;39m
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_16]
at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_16]
[2235.04 DEBUG] (Handle sendMessageAsync:183) [34m> CLOSEHANDLE@278.598[0;39m
[2235.04 INFO] (SDKD log:137) [39m at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1141
at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()[0;39m
[2235.05 WARN] (STester close:170) [33mWhile stopping driver[0;39m
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_16]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_16]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_16]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_16]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_16]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]
[2236.12 INFO] (BRun uploadFile:141) [39mUploaded RAW => http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1208/SvcRestart-HYBRID/09-11-14/025050/880a91bc80225536124352f8115cbc71.zip[0;39m
the same test passed in previous runs
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/91/consoleFull