ThrowOperationCanceledException in SendWithRetry[T](ViewQuery query) when 3 from 4 nodes restarted

http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2_win2012_64_andrei/23/consoleFull

Running SvcRestart-HYBRID: 2014-09-11 23:57:30
service/restore_delay=5
workload=HybridWorkloadGroup
service/name=SYSV_SERVICE
service/action=kill
service/count=3
testcase=ServiceFailureScenario
Logging to c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\SvcRestart-HYBRID\09-11-14\025050\880a91bc80225536124352f8115cbc71

To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
--service-restore_delay 5 --workload HybridWorkloadGroup
\ --service-name SYSV_SERVICE --service-action kill --service-count
\ 3 --testcase ServiceFailureScenario -C share\rexec --rexec_path
\ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675
\ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
\ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
\ --cluster_ssh-username=root --cluster_ssh-password=couchbase
.....

[2157.71 INFO] (Scenario run:83) RAMP phase started. Waiting for 30 seconds
[2157.71 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY 
[2157.71 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[2157.71 INFO] (SDKD log:137)  "DSType": "DSTYPE_SEEDED",
[2157.71 INFO] (SDKD log:137)  "DS": {},
[2157.71 INFO] (SDKD log:137)  "ViewParameters":

{ [2157.71 INFO] (SDKD log:137)  "limit": 0 [2157.71 INFO] (SDKD log:137)  }

,
[2157.71 INFO] (SDKD log:137)  "DesignName": "test_design",
[2157.71 INFO] (SDKD log:137)  "Options":

{ [2157.71 INFO] (SDKD log:137)  "DelayMax": 10, [2157.71 INFO] (SDKD log:137)  "DelayMin": 1, [2157.71 INFO] (SDKD log:137)  "IterWait": 1, [2157.71 INFO] (SDKD log:137)  "ViewQueryCount": -1, [2157.71 INFO] (SDKD log:137)  "ViewQueryDelay": 0, [2157.71 INFO] (SDKD log:137)  "TimeRes": 1 [2157.71 INFO] (SDKD log:137)  }

,
[2157.71 INFO] (SDKD log:137)  "ViewName": "test_view"
[2157.71 INFO] (SDKD log:137) } 
[2187.71 INFO] (Scenario run:86) Starting CHANGE phase
[2187.72 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.121.135
[2190.35 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409 
[2191.77 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@52d654
[2192.77 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.121.136
[2196.16 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@38c8c5
[2197.19 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server stop on 10.3.3.206
[2201.34 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@f522e8
[2202.34 INFO] (Scenario sleepAndDo:109) Sleeping 5 seconds
[2207.38 INFO] (Scenario executeChange:134) Turning on services for nodes
[2207.38 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.135
[2208.94 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@19c873d
[2209.96 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.121.136
[2211.47 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@12a2259
[2212.47 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start on 10.3.3.206
[2214.05 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@fe0ce9
[2215.05 INFO] (Scenario run:97) CHANGE phase done
[2215.05 INFO] (Scenario run:103) Starting REBOUND for 90 seconds
[2223.95 INFO] (SDKD log:137) 
[2223.95 INFO] (SDKD log:137) 
[2224.03 INFO] (SDKD log:137) Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: Unhandled Exception: 
[2224.04 INFO] (SDKD log:137) Unhandled Exception: System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 1
[2224.10 INFO] (SDKD log:137)  at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 0
at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (LineGobbler doFilter:115) +++ Following exception has internal ID: 3
[2224.10 INFO] (SDKD log:137)  at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1141
at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (SDKD log:137) +++ Received exception of ID 2
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (SDKD log:137) +++ Received exception of ID 2
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (SDKD log:137) +++ Received exception of ID 2
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (SDKD log:137) +++ Received exception of ID 2
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2224.10 INFO] (SDKD log:137) +++ Received exception of ID 2
[2224.10 INFO] (SDKD log:137) System.OperationCanceledException: The operation was canceled.
[2235.04 INFO] (SDKD log:137) +++ Flushing remaining exception trace buffer
[2235.04 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) ~[na:1.6.0_16]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.6.0_16]
at java.lang.Thread.run(Unknown Source) ~[na:1.6.0_16]
[2235.04 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@278.598
[2235.04 INFO] (SDKD log:137)  at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.ManualResetEventSlim.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.SpinThenBlockingWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.InternalWait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait(CancellationToken cancellationToken)
at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](ViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1141
at Couchbase.Core.Buckets.CouchbaseBucket.Query[T](IViewQuery query) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 1128
at Sdkd.Views.ViewCommand.GetViewObject(CouchbaseBucket cli, Request req, ViewOptions vo) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 164
at Sdkd.Views.ViewCommand.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\Sdkd\Views\ViewCommand.cs:line 265
at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
at SdkdConsole.Daemon.Handle.Run() in c:\jenkins\workspace\.net-sdkdclient-ng2_win2012_64_andrei\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
[2235.05 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.6.0_16]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.6.0_16]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.6.0_16]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.6.0_16]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.6.0_16]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.6.0_16]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]
[2236.12 INFO] (BRun uploadFile:141) Uploaded RAW => http://sdk-testresults.couchbase.com.s3.amazonaws.com/SDK-SDK/CB-3.0.0-1208/SvcRestart-HYBRID/09-11-14/025050/880a91bc80225536124352f8115cbc71.zip

the same test passed in previous runs
http://sdkbuilds.couchbase.com/job/.net-sdkdclient-ng2-andrei/91/consoleFull