c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args -d all:debug --no-upload --dsw_kvcount 10 [0,13 INFO] (BRun run:435) Initializing history database ============================================================ Running Rb2Out-KV rebalance/mode=out rebalance/count=2 testcase=RebalanceScenario Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1035\Rb2Out-KV\07-30-14\054997\a2b69f611caf7cb178676c3791c464ba To re-run the test, copy/paste the following into the shell. You may also copy/paste (except the first line) into an argfile 8<---------------------------------------- ./stester \ \ --rebalance-mode out --rebalance-count 2 --testcase \ RebalanceScenario -C share\rexec --rexec_path \ C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port 8675 \ --cluster_node=10.3.121.134 --cluster_node=10.3.121.135 \ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206 \ --cluster_ssh-username=root --cluster_ssh-password=couchbase --dsw_kvcount 10 ---------------------------------------->8 [1,38 WARN] (Drivers getDriver:76) 'rexec' is now mapped to *local* execution only. Use RemoteExecutingDriver for remote execution [1,62 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds. [1,62 INFO] (RunContext run:124) Starting cluster and driver [1,63 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe' [1,63 INFO] (CBCluster startCluster:360) Node http://10.3.121.134:8091 is master now [1,63 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations.. [1,80 INFO] (SDKD log:137) --- Logging Self-Test --- [1,81 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message [1,81 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message [1,81 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message [1,81 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message [1,82 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675 [2,17 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [2,17 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0 [2,28 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true},"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}} [8,99 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node [9,50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84] [9,50 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84] [12,37 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.136:8091 [12,37 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.135:8091 [12,37 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.3.206:8091 [12,38 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.134:8091 [12,39 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.135 [12,39 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.136 [12,39 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.134 [12,39 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.3.206 [13,26 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@46dd7463 [13,26 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@750433ed [13,27 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@66ed489f [13,27 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@58e58e3d [13,27 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@542487b1 [20,22 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.135:8091 [38,52 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.136:8091 [43,26 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.3.206:8091 [50,08 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance [51,05 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete [51,06 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups [51,30 INFO] (CBCluster setupMainBucket:209) Creating bucket default [51,55 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted [56,12 INFO] (CBCluster waitForBucketReady:203) Bucket creation done [59,76 INFO] (RunContext run:143) Driver and cluster initialized [60,01 INFO] (RunContext call:167) Running scenario.. [60,01 INFO] (Scenario run:72) Starting RAMP phase [60,01 INFO] (Workload call:112) Preloading items into cluster before workload [60,02 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [60,02 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [60,03 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101 [60,04 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [60,06 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net [60,06 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e [60,08 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net [60,08 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821 [64,13 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1 [64,13 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=false, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=0, DelayMin=0, IterWait=100, TimeRe s=0}} [67,95 DEBUG] (Handle receiveMessage:158) < MC_DS_MUTATE_SET@101.2 => {[OK]: 10} [67,95 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3 [67,95 INFO] (SDKD log:137) ReleaseClient [67,96 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [67,96 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.5 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,96 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,96 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@111.13 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,96 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@110.12 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@109.11 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.10 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.9 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,97 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.6 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,98 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [67,98 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.8 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,98 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.7 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [67,99 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [67,99 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [68,00 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [68,00 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [68,00 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [68,00 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103 [68,00 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [68,01 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [68,01 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [68,01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102 [68,01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 111 [68,01 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108 [68,02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 110 [68,02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107 [68,02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 109 [68,02 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104 [68,03 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106 [68,03 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105 [71,77 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [71,77 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.5 [71,77 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@103.14 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes =1}} [71,78 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [71,79 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@112.15 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [71,79 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 112 [75,54 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [75,54 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4 [75,55 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@102.16 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [75,55 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [75,55 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@113.17 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [75,56 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 113 [76,15 INFO] (SDKD log:137) !!!!!!!!!!!!!!!!System.Security.Authentication.AuthenticationException False [76,16 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409 [79,32 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [79,32 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@111.13 [79,32 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@111.18 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes =1}} [79,33 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [79,33 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@114.19 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [79,34 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 114 [83,21 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [83,21 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.10 [83,22 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.20 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes =1}} [83,23 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [83,23 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@115.21 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [83,23 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 115 [87,07 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [87,07 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@110.12 [87,08 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@110.22 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [87,08 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [87,08 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@116.23 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [87,09 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 116 [90,85 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [90,85 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.9 [90,85 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@107.24 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [90,86 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [90,86 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@117.25 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [90,86 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 117 [94,76 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [94,76 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@109.11 [94,76 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@109.26 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes =1}} [94,77 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [94,77 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@118.27 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [94,77 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 118 [98,54 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [98,54 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.6 [98,54 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@104.28 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [98,55 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [98,55 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@119.29 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [98,56 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 119 [102,32 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [102,32 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.8 [102,32 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@106.30 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [102,33 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [102,33 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@120.31 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [102,34 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 120 [106,10 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [106,10 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.7 [106,10 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@105.32 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRe s=1}} [106,11 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle [106,11 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@121.33 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134} [106,12 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 121 [110,21 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [110,21 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@112.15 [110,21 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@112.34 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRe s=1}} [117,26 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [117,26 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@113.17 [117,26 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@113.35 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [117,74 INFO] (SDKD log:137) !!!!!!!!!!!!!!!!System.Security.Authentication.AuthenticationException False [120,58 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [120,58 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@114.19 [120,58 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@114.36 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRe s=1}} [124,36 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [124,36 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@115.21 [124,36 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@115.37 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [128,14 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [128,14 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@116.23 [128,15 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@116.38 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRe s=1}} [131,93 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [131,94 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@117.25 [131,94 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@117.39 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [135,71 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [135,71 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@118.27 [135,72 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@118.40 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRe s=1}} [139,53 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [139,53 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@119.29 [139,54 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@119.41 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [146,30 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Shared clients disabled. Creating anew [146,30 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@120.31 [146,30 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@120.42 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRe s=1}} [147,98 INFO] (SDKD log:137) !!!!!!!!!!!!!!!!System.Security.Authentication.AuthenticationException False [153,79 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@121.33 [153,79 DEBUG] (Handle sendMessageAsync:183) > MC_DS_GET@121.43 => {DSType=DSTYPE_SEEDED, DS={KSeed=SimpleKey, VSeed=SimpleValue, Count=10, Continuous=true, VSize=128, Repeat=REP, KSize=32}, Options={DelayMax=10, DelayMin=1, IterWait=1, TimeRes=1}} [153,79 INFO] (Scenario run:83) RAMP phase started. Waiting for 30 seconds [183,79 INFO] (Scenario run:86) Starting CHANGE phase [184,28 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% [185,53 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% [186,79 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% [188,03 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% [189,27 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 0% [190,51 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 3% [191,77 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 4% [193,01 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 10% [193,85 INFO] (SDKD log:137) [193,85 INFO] (SDKD log:137) unhandled exception: System.Collections.Generic.KeyNotFoundException: This key is not in the dictionary. [193,86 INFO] (SDKD log:137) in System.Collections.Concurrent.ConcurrentDictionary`2.get_Item(TKey key) [193,86 INFO] (SDKD log:137) in Couchbase.Configuration.Server.Providers.Streaming.HttpStreamingProvider.ConfigChangedHandler(IBucketConfig bucketConfig) in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\ Streaming\HttpStreamingProvider.cs:line 151 [193,86 INFO] (SDKD log:137) in Couchbase.Configuration.Server.Providers.Streaming.ConfigThreadState.ListenForConfigChanges() in c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\Providers\Streaming\ConfigThreadState. cs:line 104 [193,87 INFO] (SDKD log:137) in System.Threading.ThreadHelper.ThreadStart_Context(Object state) [193,87 INFO] (SDKD log:137) in System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [193,87 INFO] (SDKD log:137) in System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [193,87 INFO] (SDKD log:137) in System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) [193,87 INFO] (SDKD log:137) in System.Threading.ThreadHelper.ThreadStart() [194,26 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 16% [195,50 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17% [196,74 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17% [197,98 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17% [199,14 ERROR] (BRun maybePropagate:237) Suppressing driver-related error com.couchbase.sdkdclient.context.HarnessException: Driver crashed at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_60] at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_60] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0_60] at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_60] [199,17 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.44 [199,17 WARN] (STester close:170) While stopping driver java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_60] at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_60] at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_60] at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.7.0_60] at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.7.0_60] at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.7.0_60] at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.7.0_60] at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.7.0_60] at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.7.0_60] at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na] at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na] [199,22 DEBUG] (RebalanceWaiter sweepOnce:38) Rebalance Progress: 17% c:\jenkins\workspace-net2\sdkdclient-ng>