Critical Description
Unfortunately this run was without client logging because I was not planning to test client)
will try to provide them later, if I will be able to reproduce it
c:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,13 INFO] (BRun run:441) Initializing history database
============================================================
Running Rb2Out-HYBRID: 2014-09-06 12:43:12
rebalance/count=2
workload=HybridWorkloadGroup
rebalance/mode=out
testcase=RebalanceScenario
Logging to c:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1208\Rb2Out-HYBRID\09-06-14\034992\74769ea477613d9a56c4eb5d41d6c987
To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
\ --rebalance-count 2 --workload HybridWorkloadGroup
\ --rebalance-mode out --testcase RebalanceScenario -C share\rexec
\ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
\ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
\ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
\ --cluster_useSSL=True --cluster_ssh-username=root
--cluster_ssh-password=couchbase
.....
[3870,95 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[3870,95 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 127
[3876,33 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@127.56
[3876,33 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@127.57 => {Options=
est_view, DS={}}
[3876,34 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[3876,34 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@128.58 => {Options={OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]], Username=default, SSL=true, Password=}, Bucket=default, Port=8091, Hostname=10.3.121.134}
[3876,34 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[3876,34 INFO] (SDKD log:137) "Options": { [3876,35 INFO] (SDKD log:137) "DelayMin": 1, [3876,35 INFO] (SDKD log:137) "TimeRes": 1, [3876,35 INFO] (SDKD log:137) "ViewQueryCount": -1, [3876,35 INFO] (SDKD log:137) "IterWait": 1, [3876,36 INFO] (SDKD log:137) "ViewQueryDelay": 0, [3876,36 INFO] (SDKD log:137) "DelayMax": 10 [3876,36 INFO] (SDKD log:137) },
[3876,36 INFO] (SDKD log:137) "ViewParameters": { [3876,37 INFO] (SDKD log:137) "limit": 0 [3876,37 INFO] (SDKD log:137) },
[3876,37 INFO] (SDKD log:137) "DesignName": "test_design",
[3876,37 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[3876,38 INFO] (SDKD log:137) "ViewName": "test_view",
[3876,38 INFO] (SDKD log:137) "DS": {}
[3876,38 INFO] (SDKD log:137) }
[3876,38 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[3876,39 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 128
[3892,59 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@128.58
[3892,59 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@128.59 => {Options={DelayMin=1, TimeRes=1, ViewQueryCount=-1, IterWait=1, ViewQueryDelay=0, DelayMax=10}
, ViewParameters=
{limit=0}, DesignName=test_design, DSType=DSTYPE_SEEDED, ViewName=t
est_view, DS={}}
[3892,59 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_QUERY
[3892,59 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[3892,59 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@129.60 => {Options=
[3892,59 INFO] (SDKD log:137) "Options": { [3892,59 INFO] (SDKD log:137) "DelayMin": 1, [3892,60 INFO] (SDKD log:137) "TimeRes": 1, [3892,60 INFO] (SDKD log:137) "ViewQueryCount": -1, [3892,60 INFO] (SDKD log:137) "IterWait": 1, [3892,60 INFO] (SDKD log:137) "ViewQueryDelay": 0, [3892,60 INFO] (SDKD log:137) "DelayMax": 10 [3892,60 INFO] (SDKD log:137) },
[3892,60 INFO] (SDKD log:137) "ViewParameters": { [3892,60 INFO] (SDKD log:137) "limit": 0 [3892,61 INFO] (SDKD log:137) },
[3892,61 INFO] (SDKD log:137) "DesignName": "test_design",
[3892,61 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[3892,61 INFO] (SDKD log:137) "ViewName": "test_view",
[3892,61 INFO] (SDKD log:137) "DS": {}
[3892,62 INFO] (SDKD log:137) }
[3892,62 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[3892,62 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 129
[3898,85 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@129.60
[3898,85 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@129.61 => {Options={DelayMin=0, TimeRes=0, IterWait=1, DelayMax=0}, Schema={InflateLevel=40, InflateContent=meh}, DSType=DSTYPE_SEEDED, DS={KSize=12, KSeed=ViewFillerSeed, VSize=512, VSeed=
ViewFillerSeed, Continuous=false, Repeat=rep, Count=1000}}
[3898,85 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CB_VIEW_LOAD
[3898,86 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {
[3898,86 INFO] (SDKD log:137) "Options": { [3898,86 INFO] (SDKD log:137) "DelayMin": 0, [3898,86 INFO] (SDKD log:137) "TimeRes": 0, [3898,86 INFO] (SDKD log:137) "IterWait": 1, [3898,87 INFO] (SDKD log:137) "DelayMax": 0 [3898,87 INFO] (SDKD log:137) },
[3898,87 INFO] (SDKD log:137) "Schema": { [3898,87 INFO] (SDKD log:137) "InflateLevel": 40, [3898,88 INFO] (SDKD log:137) "InflateContent": "meh" [3898,88 INFO] (SDKD log:137) },
[3898,88 INFO] (SDKD log:137) "DSType": "DSTYPE_SEEDED",
[3898,88 INFO] (SDKD log:137) "DS": { [3898,88 INFO] (SDKD log:137) "KSize": 12, [3898,89 INFO] (SDKD log:137) "KSeed": "ViewFillerSeed", [3898,89 INFO] (SDKD log:137) "VSize": 512, [3898,89 INFO] (SDKD log:137) "VSeed": "ViewFillerSeed", [3898,89 INFO] (SDKD log:137) "Continuous": false, [3898,89 INFO] (SDKD log:137) "Repeat": "rep", [3898,89 INFO] (SDKD log:137) "Count": 1000 [3898,90 INFO] (SDKD log:137) }
[3898,90 INFO] (SDKD log:137) }
[4038,12 INFO] (SDKD log:137) [Sdkd.ViewLoader|Warn] there are no OK responses in ViewLoader
[4038,12 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {
[4038,12 INFO] (SDKD log:137) "0": 579,
[4038,12 DEBUG] (Handle receiveMessage:158) < CB_VIEW_LOAD@129.61 => {[OK]: 579, [MEMD:GENERIC]: 421}
[4038,12 INFO] (SDKD log:137) "264": 421
[4038,13 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@129.62
[4038,13 INFO] (SDKD log:137) }
[4038,13 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@130.63 => {Options={OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]], Username=default, SSL=true, Password=}
, Bucket=default, Port=8091, Hostname=10.3.121.134}
[4038,13 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[4038,13 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Got command CLOSEHANDLE
[4038,14 INFO] (SDKD log:137) [Sdkd.Handle|Warn] Command data: {}
[4038,14 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket -436020959
[4038,14 INFO] (SDKD log:137) [Sdkd.Handle|Warn] DispatchCommand(req, SdkHandle) == false
[4038,14 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 130
[4039,99 INFO] (SDKD log:137)
[4040,07 INFO] (SDKD log:137) Unhandled Exception: System.ArgumentOutOfRangeException: Index and the counter should indicate the position at the buffer
[4040,08 INFO] (SDKD log:137) prameter name: bytes
[4040,08 INFO] (SDKD log:137) at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count)
[4040,08 INFO] (SDKD log:137) at Couchbase.Core.Serializers.TypeSerializer.Deserialize(Byte[] buffer, Int32 offset, Int32 length) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\Core\Serializers\TypeSerializer.cs:line 197
[4040,08 INFO] (SDKD log:137) at Couchbase.Core.Serializers.TypeSerializer.Deserialize[T](Byte[] buffer, Int32 offset, Int32 length) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\Core\Serializers\TypeSerializer.cs:line 86
[4040,08 INFO] (SDKD log:137) at Couchbase.IO.Operations.OperationBase`1.GetValue() at d:\couchbase_src\couchbase-net-client\Src\Couchbase\IO\Operations\OperationBase'.cs:line 229
[4040,09 INFO] (SDKD log:137) at Couchbase.IO.Operations.OperationBase`1.GetResult() at d:\couchbase_src\couchbase-net-client\Src\Couchbase\IO\Operations\OperationBase'.cs:line 198
[4040,09 INFO] (SDKD log:137) at Couchbase.Core.Server.Send[T](IOperation`1 operation) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\Core\Server.cs:line 91
[4040,09 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.SendWithRetry[T](IOperation`1 operation) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 98
[4040,09 INFO] (SDKD log:137) at Couchbase.Core.Buckets.CouchbaseBucket.Get[T](String key) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\Core\Buckets\CouchbaseBucket.cs:line 894
[4040,09 INFO] (SDKD log:137) at Sdkd.Commands.GetCommand.ExecuteSingle(String key, String value) at d:\couchbase_src\sdkd-net\src\Sdkd\Commands\GetCommand.cs:line 17
[4040,10 INFO] (SDKD log:137) at Sdkd.Commands.KVCommand.Run() at d:\couchbase_src\sdkd-net\src\Sdkd\Commands\KVCommand.cs:line 24
[4040,10 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.DispatchCommand(Request req, CouchbaseBucket cli) at d:\couchbase_src\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 266
[4040,10 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.Run() at d:\couchbase_src\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 146
[4040,10 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[4040,10 INFO] (SDKD log:137) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[4040,10 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[4040,11 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[4040,11 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart()
[4078,01 INFO] (SDKD log:137)
[4078,01 INFO] (SDKD log:137) Unhandled Exception: System.NullReferenceException: Object reference not set to an instance of the object.
[4078,02 INFO] (SDKD log:137) at Couchbase.Core.ClusterManager.CreateBucket(String bucketName, String password) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 160
[4078,02 INFO] (SDKD log:137) at Couchbase.CouchbaseCluster.OpenBucket(String bucketname, String password) at d:\couchbase_src\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 84
[4078,02 INFO] (SDKD log:137) at SdkdConsole.Daemon.Handle.Run() at d:\couchbase_src\sdkd-net\src\SdkdConsole\Daemon\Handle.cs:line 125
[4078,02 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
[4078,02 INFO] (SDKD log:137) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[4078,03 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[4078,03 INFO] (SDKD log:137) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[4078,03 INFO] (SDKD log:137) at System.Threading.ThreadHelper.ThreadStart()
[4090,68 ERROR] (BRun maybePropagate:243) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.8.0_11]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.8.0_11]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.8.0_11]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.8.0_11]
at java.lang.Thread.run(Unknown Source) ~[na:1.8.0_11]
[4090,70 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.64
[4090,71 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_11]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.8.0_11]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.8.0_11]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.8.0_11]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.8.0_11]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.8.0_11]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.8.0_11]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.8.0_11]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.8.0_11]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:287) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:394) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:478) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:510) [sdkdclient-1.0-SNAPSHOT.jar:na]