System.ArgumentOutOfRangeException in CanRetryOperation in case Timed out error/network issues

SHA: 940183971ea8d0570033ed28bba2e7c84ddc2b71

I run the tests through the vpn and sdkd tests probably are not ideal but...

c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,14 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
rebalance/mode=out
workload=HybridWorkloadGroup
rebalance/count=2
testcase=RebalanceScenario
Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1105\Rb2Out-HYBRID\08-06-14\047299\d8869f6da67dff794890340838f386d3

To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
\ --rebalance-mode out --workload HybridWorkloadGroup
\ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
\ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
\ 8675 --cluster_node=10.3.4.144 --cluster_node=10.3.4.145
\ --cluster_node=10.3.4.146 --cluster_node=10.3.4.147
--cluster_ssh-username=root --cluster_ssh-password=couchbase
---------------------------------------->8
[1,57 WARN] (Drivers getDriver:76) 'rexec' is now mapped to local execution only. Use RemoteExecutingDriver for remote execution
[1,91 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,92 INFO] (RunContext run:124) Starting cluster and driver
[1,92 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,92 INFO] (CBCluster startCluster:360) Node http://10.3.4.144:8091 is master now
[1,92 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[2,09 INFO] (SDKD log:137) — Logging Self-Test —
[2,09 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[2,09 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[2,09 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[2,10 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[2,10 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,44 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,44 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,57 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":

{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true}

,"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[9,02 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.145:8091,ns_1@10.3.4.145>
[9,51 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.146:8091,ns_1@10.3.4.146>
[10,00 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.4.147:8091,ns_1@10.3.4.147>
[10,50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[10,50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[12,56 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.145:8091
[12,56 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.144:8091
[12,57 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.147:8091
[12,57 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.4.146:8091
[12,58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.146
[12,58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.145
[12,58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.147
[12,58 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.4.144
[13,42 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@54db08f1
[13,42 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@782831bc
[13,42 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@13d3589e
[13,43 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@2b9d24b
[13,43 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@2548d5b2
[22,50 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.145:8091
[28,06 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.146:8091
[37,69 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.4.147:8091
[42,32 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[43,29 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[43,29 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[43,54 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[43,79 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[52,92 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[55,76 INFO] (RunContext run:143) Driver and cluster initialized
[56,01 INFO] (RunContext call:167) Running scenario..
[56,01 INFO] (Scenario run:72) Starting RAMP phase
[56,01 INFO] (Workload setupDesign:63) Creating design test_design
[57,28 INFO] (Workload setupDesign:80) Design creation done
[57,29 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[57,29 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}, Hostname=10.3.4.144}
[57,30 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[57,31 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[57,33 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[57,33 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[57,36 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[57,36 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[64,97 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[64,98 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}, Schema={InflateContent=meh, InflateLevel=40}, O
ptions={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[64,98 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[64,99 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[64,99 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[64,99 INFO] (SDKD log:137) "DS": { [64,99 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed", [64,99 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed", [65,00 INFO] (SDKD log:137) "Count": 1000, [65,00 INFO] (SDKD log:137) "Continuous": false, [65,00 INFO] (SDKD log:137) "VSize": 512, [65,00 INFO] (SDKD log:137) "Repeat": "rep", [65,00 INFO] (SDKD log:137) "KSize": 12 [65,01 INFO] (SDKD log:137) },
[65,01 INFO] (SDKD log:137) "Schema": { [65,01 INFO] (SDKD log:137) "InflateContent": "meh", [65,01 INFO] (SDKD log:137) "InflateLevel": 40 [65,02 INFO] (SDKD log:137) },
[65,02 INFO] (SDKD log:137) "Options": { [65,02 INFO] (SDKD log:137) "DelayMax": 0, [65,03 INFO] (SDKD log:137) "DelayMin": 0, [65,03 INFO] (SDKD log:137) "IterWait": 1, [65,03 INFO] (SDKD log:137) "TimeRes": 0 [65,03 INFO] (SDKD log:137) }
[65,04 INFO] (SDKD log:137) }
[301,88 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[301,89 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@101.2
[301,89 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[301,89 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[301,89 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}

, Hostname=10.3.4.144}
[301,89 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[301,90 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[301,90 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -788777045
[301,90 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()-788777045
[301,90 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[301,91 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[301,91 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[308,11 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[308,11 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@102.5 => {DSType=DSTYPE_SEEDED, DS=

{KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}

, Schema=

{InflateContent=meh, InflateLevel=40}

,
Options={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[308,12 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[308,12 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[308,12 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[308,12 INFO] (SDKD log:137) "DS":

{ [308,12 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed", [308,13 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed", [308,13 INFO] (SDKD log:137) "Count": 1000, [308,13 INFO] (SDKD log:137) "Continuous": false, [308,14 INFO] (SDKD log:137) "VSize": 512, [308,14 INFO] (SDKD log:137) "Repeat": "rep", [308,14 INFO] (SDKD log:137) "KSize": 12 [308,14 INFO] (SDKD log:137) }

,
[308,14 INFO] (SDKD log:137) "Schema":

{ [308,14 INFO] (SDKD log:137) "InflateContent": "meh", [308,15 INFO] (SDKD log:137) "InflateLevel": 40 [308,15 INFO] (SDKD log:137) }

,
[308,15 INFO] (SDKD log:137) "Options":

{ [308,15 INFO] (SDKD log:137) "DelayMax": 0, [308,15 INFO] (SDKD log:137) "DelayMin": 0, [308,16 INFO] (SDKD log:137) "IterWait": 1, [308,16 INFO] (SDKD log:137) "TimeRes": 0 [308,16 INFO] (SDKD log:137) }

[308,17 INFO] (SDKD log:137) }
[510,64 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {}
[510,64 DEBUG] (Handle receiveMessage:158) < [SDKD:SDKD_EINVAL="Need at least one kv pair. None requested"] CB_VIEW_LOAD@102.5
[510,65 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[510,65 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[510,65 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}, Hostname=10.3.4.144}
[510,65 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[510,65 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -788777045
[510,65 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket client.Dispose()-788777045
[510,66 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[510,66 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[510,66 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[510,66 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[516,85 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[516,86 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@103.8 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=tes
t_view}
[516,86 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[516,87 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[516,87 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.9 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}

, Hostname=10.3.4.144}
[516,87 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[516,87 INFO] (SDKD log:137) "DS": {},
[516,87 INFO] (SDKD log:137) "ViewParameters": {},
[516,87 INFO] (SDKD log:137) "DesignName": "test_design",
[516,88 INFO] (SDKD log:137) "Options":

{ [516,88 INFO] (SDKD log:137) "DelayMax": 10, [516,88 INFO] (SDKD log:137) "DelayMin": 1, [516,88 INFO] (SDKD log:137) "IterWait": 1, [516,89 INFO] (SDKD log:137) "ViewQueryCount": -1, [516,89 INFO] (SDKD log:137) "ViewQueryDelay": 0, [516,89 INFO] (SDKD log:137) "TimeRes": 1 [516,89 INFO] (SDKD log:137) }

,
[516,89 INFO] (SDKD log:137) "ViewName": "test_view"
[516,89 INFO] (SDKD log:137) }
[516,90 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[516,90 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[516,90 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[522,58 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.9
[522,58 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@104.10 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options=

{DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=te
st_view}
[522,59 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[522,59 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.11 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}, Hostname=10.3.4.144}
[522,59 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[522,59 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[522,59 INFO] (SDKD log:137) "DS": {},
[522,60 INFO] (SDKD log:137) "ViewParameters": {},
[522,60 INFO] (SDKD log:137) "DesignName": "test_design",
[522,61 INFO] (SDKD log:137) "Options": { [522,61 INFO] (SDKD log:137) "DelayMax": 10, [522,61 INFO] (SDKD log:137) "DelayMin": 1, [522,61 INFO] (SDKD log:137) "IterWait": 1, [522,61 INFO] (SDKD log:137) "ViewQueryCount": -1, [522,61 INFO] (SDKD log:137) "ViewQueryDelay": 0, [522,62 INFO] (SDKD log:137) "TimeRes": 1 [522,62 INFO] (SDKD log:137) },
[522,62 INFO] (SDKD log:137) "ViewName": "test_view"
[522,62 INFO] (SDKD log:137) }
[522,63 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[522,63 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[522,63 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[528,32 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.11
[528,32 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@105.12 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}

, ViewName=te
st_view}
[528,33 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[528,33 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[528,33 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.13 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}, Hostname=10.3.4.144}
[528,34 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[528,34 INFO] (SDKD log:137) "DS": {},
[528,34 INFO] (SDKD log:137) "ViewParameters": {},
[528,34 INFO] (SDKD log:137) "DesignName": "test_design",
[528,35 INFO] (SDKD log:137) "Options": { [528,35 INFO] (SDKD log:137) "DelayMax": 10, [528,35 INFO] (SDKD log:137) "DelayMin": 1, [528,35 INFO] (SDKD log:137) "IterWait": 1, [528,35 INFO] (SDKD log:137) "ViewQueryCount": -1, [528,36 INFO] (SDKD log:137) "ViewQueryDelay": 0, [528,36 INFO] (SDKD log:137) "TimeRes": 1 [528,36 INFO] (SDKD log:137) },
[528,36 INFO] (SDKD log:137) "ViewName": "test_view"
[528,36 INFO] (SDKD log:137) }
[528,37 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[528,37 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[528,37 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[533,76 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.13
[533,76 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@106.14 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=te
st_view}
[533,76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[533,76 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[533,77 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.15 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}

, Hostname=10.3.4.144}
[533,77 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[533,77 INFO] (SDKD log:137) "DS": {},
[533,77 INFO] (SDKD log:137) "ViewParameters": {},
[533,77 INFO] (SDKD log:137) "DesignName": "test_design",
[533,77 INFO] (SDKD log:137) "Options":

{ [533,78 INFO] (SDKD log:137) "DelayMax": 10, [533,78 INFO] (SDKD log:137) "DelayMin": 1, [533,78 INFO] (SDKD log:137) "IterWait": 1, [533,78 INFO] (SDKD log:137) "ViewQueryCount": -1, [533,78 INFO] (SDKD log:137) "ViewQueryDelay": 0, [533,78 INFO] (SDKD log:137) "TimeRes": 1 [533,79 INFO] (SDKD log:137) }

,
[533,79 INFO] (SDKD log:137) "ViewName": "test_view"
[533,79 INFO] (SDKD log:137) }
[533,79 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[533,79 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[533,80 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[539,57 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.15
[539,57 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@107.16 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options=

{DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}

, ViewName=te
st_view}
[539,58 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[539,58 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[539,58 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[539,58 INFO] (Workload call:112) Preloading items into cluster before workload
[539,59 INFO] (SDKD log:137) "DS": {},
[539,59 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.17 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.4.145",8091],["10.3.4.146",8091],["10.3.4.147",8091]]}

, Hostname=10.3.4.144}
[539,59 INFO] (SDKD log:137) "ViewParameters": {},
[539,59 INFO] (SDKD log:137) "DesignName": "test_design",
[539,59 INFO] (SDKD log:137) "Options":

{ [539,60 INFO] (SDKD log:137) "DelayMax": 10, [539,60 INFO] (SDKD log:137) "DelayMin": 1, [539,60 INFO] (SDKD log:137) "IterWait": 1, [539,60 INFO] (SDKD log:137) "ViewQueryCount": -1, [539,61 INFO] (SDKD log:137) "ViewQueryDelay": 0, [539,61 INFO] (SDKD log:137) "TimeRes": 1 [539,61 INFO] (SDKD log:137) }

,
[539,61 INFO] (SDKD log:137) "ViewName": "test_view"
[539,61 INFO] (SDKD log:137) }
[539,62 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[539,62 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[539,62 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew
[544,94 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.17
[544,94 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.18 => {DSType=DSTYPE_SEEDED, DS=

{KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=false, VSize=128, Repeat=REP, KSize=32}

, Options={DelayMax=0, DelayMin=0, IterWait=100, T
imeRes=0}}
[544,95 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command MC_DS_MUTATE_SET
[544,95 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[544,95 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[544,95 INFO] (SDKD log:137) "DS":

{ [544,96 INFO] (SDKD log:137) "KSeed": "SimpleKey", [544,96 INFO] (SDKD log:137) "VSeed": "SimpleValue", [544,96 INFO] (SDKD log:137) "Count": 15000, [544,96 INFO] (SDKD log:137) "Continuous": false, [544,96 INFO] (SDKD log:137) "VSize": 128, [544,97 INFO] (SDKD log:137) "Repeat": "REP", [544,97 INFO] (SDKD log:137) "KSize": 32 [544,97 INFO] (SDKD log:137) }

,
[544,97 INFO] (SDKD log:137) "Options":

{ [544,97 INFO] (SDKD log:137) "DelayMax": 0, [544,98 INFO] (SDKD log:137) "DelayMin": 0, [544,98 INFO] (SDKD log:137) "IterWait": 100, [544,98 INFO] (SDKD log:137) "TimeRes": 0 [544,98 INFO] (SDKD log:137) }

[544,98 INFO] (SDKD log:137) }
[547,24 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[573,65 INFO] (SDKD log:137)
[573,66 INFO] (SDKD log:137) Необработанное исключение: System.ArgumentOutOfRangeException: Заданный аргумент находится вне диапазона допустимых значений.
[573,66 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.CanRetryOperation[T](IOperationResult`1 operationResult, IOperation`1 operation, IServer server) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\Couc
hbaseBucket.cs:line 180
[573,66 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 102
[573,66 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.Upsert[T](String key, T value) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 272
[573,67 INFO] (SDKD log:137) at Sdkd.Commands.SetCommand.ExecuteSingle(String key, String value) at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\SetCommand.cs:line 77
[573,67 INFO] (SDKD log:137) at Sdkd.Commands.KVCommand.Run() at c:\jenkins\workspace-net2\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 25
[573,67 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) at c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 231
[573,67 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.Run() at c:\jenkins\workspace-net2\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 153
[573,68 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[573,68 INFO] (SDKD log:137) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[573,68 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[573,68 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[573,69 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart()
[3758,06 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0_60]
at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_60]
[3758,09 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.19
[3758,09 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_60]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_60]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.7.0_60]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.7.0_60]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.7.0_60]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]

please take a look on client logs related to SimpleKeyREP51REP51REP51REP51REP51REP51REP51 key

2014-08-06 16:17:50,486 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 74bf91e3-557b-471a-9273-6e836b9bb1d2 on 10.3.4.145:11210 - b61217ce-5d73-411f-b9f6-d7f55928c393
2014-08-06 16:17:50,486 [27] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation doesn't support retries for key SimpleKeyREP50REP50REP50REP50REP50REP50REP50
2014-08-06 16:17:50,486 [27] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP50REP50REP50REP50REP50REP50REP50 failed after 0 retries. Reason: Timed out.
2014-08-06 16:17:50,486 [27] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#44 on thread 27
2014-08-06 16:17:50,486 [27] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 180 for key SimpleKeyREP51REP51REP51REP51REP51REP51REP51 - rev0
2014-08-06 16:17:50,486 [27] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP51REP51REP51REP51REP51REP51REP51 using server 10.3.4.144:11210
2014-08-06 16:17:50,486 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: 840581c8-e22d-43c8-b7fb-e0b43cdff84c | 10.3.4.144:11210 | [0, 2] - 4596dec1-2c02-41bc-a1ef-ae9b058d0fcd - Disposed: False
2014-08-06 16:17:50,486 [27] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP51REP51REP51REP51REP51REP51REP51 using 180 on 10.3.4.144:11210
2014-08-06 16:17:50,661 [12] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:50,695 [22] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.147:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:50,927 [21] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:50,928 [26] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.147:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:50,928 [12] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.145:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:50,932 [17] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:50,987 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: 840581c8-e22d-43c8-b7fb-e0b43cdff84c on 10.3.4.144:11210 - 4596dec1-2c02-41bc-a1ef-ae9b058d0fcd
2014-08-06 16:17:50,987 [27] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation doesn't support retries for key SimpleKeyREP51REP51REP51REP51REP51REP51REP51
2014-08-06 16:17:50,987 [27] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP51REP51REP51REP51REP51REP51REP51 failed after 0 retries. Reason: Timed out.
2014-08-06 16:17:50,987 [27] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#44 on thread 27
2014-08-06 16:17:50,987 [27] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 354 for key SimpleKeyREP52REP52REP52REP52REP52REP52REP52 - rev0
2014-08-06 16:17:50,987 [27] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP52REP52REP52REP52REP52REP52REP52 using server 10.3.4.147:11210
2014-08-06 16:17:50,987 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: dddae2e9-fcca-44a7-a96b-6fb075d86c8e | 10.3.4.147:11210 | [0, 2] - 35f5a35e-087a-421d-a8ed-2d450d3b1cc2 - Disposed: False
2014-08-06 16:17:50,987 [27] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP52REP52REP52REP52REP52REP52REP52 using 354 on 10.3.4.147:11210
2014-08-06 16:17:51,169 [21] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.145:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,173 [22] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.145:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,180 [17] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,419 [26] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,424 [12] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,440 [17] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.147:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,487 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: dddae2e9-fcca-44a7-a96b-6fb075d86c8e on 10.3.4.147:11210 - 35f5a35e-087a-421d-a8ed-2d450d3b1cc2
2014-08-06 16:17:51,487 [27] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation doesn't support retries for key SimpleKeyREP52REP52REP52REP52REP52REP52REP52
2014-08-06 16:17:51,487 [27] DEBUG Couchbase.Core.Buckets.CouchbaseBucket - Operation for key SimpleKeyREP52REP52REP52REP52REP52REP52REP52 failed after 0 retries. Reason: Timed out.
2014-08-06 16:17:51,487 [27] INFO Couchbase.Configuration.ConfigContextBase - Getting KeyMapper for rev#44 on thread 27
2014-08-06 16:17:51,487 [27] INFO Couchbase.Core.Buckets.VBucketKeyMapper - Using index 255 for key SimpleKeyREP53REP53REP53REP53REP53REP53REP53 - rev0
2014-08-06 16:17:51,487 [27] DEBUG Couchbase.Core.Server - Sending SimpleKeyREP53REP53REP53REP53REP53REP53REP53 using server 10.3.4.147:11210
2014-08-06 16:17:51,487 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Acquire existing: dddae2e9-fcca-44a7-a96b-6fb075d86c8e | 10.3.4.147:11210 | [0, 2] - 35f5a35e-087a-421d-a8ed-2d450d3b1cc2 - Disposed: False
2014-08-06 16:17:51,487 [27] INFO Couchbase.IO.ConnectionBase - Sending key SimpleKeyREP53REP53REP53REP53REP53REP53REP53 using 255 on 10.3.4.147:11210
2014-08-06 16:17:51,670 [21] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,671 [22] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,677 [12] DEBUG Couchbase.Views.ViewQuery - http://10.3.4.144:8092/default/_design/test_design/_view/test_view?
2014-08-06 16:17:51,743 [27] DEBUG Couchbase.IO.ConnectionPool`1 - Releasing: dddae2e9-fcca-44a7-a96b-6fb075d86c8e on 10.3.4.147:11210 - 35f5a35e-087a-421d-a8ed-2d450d3b1cc2

cleint logs are attached