Collection was modified; enumeration operation may not execute in ConfigContextBase.Dispose

sdkd-net is still under development to fix http://www.couchbase.com/issues/browse/NCBC-567

but i think such problems should not occur on the client side

C:\jenkins\workspace-net2\sdkdclient-ng>call packages\sdkdclient\bin\brun.bat -I cluster_config.ini -I sdkd.args --variants HYBRID -d all:debug --no-upload
[0,14 INFO] (BRun run:435) Initializing history database

============================================================
Running Rb2Out-HYBRID
rebalance/mode=out
workload=HybridWorkloadGroup
rebalance/count=2
testcase=RebalanceScenario
Logging to C:\jenkins\workspace-net2\sdkdclient-ng\log\SDK-SDK\CB-3.0.0-1060\Rb2Out-HYBRID\08-05-14\040766\93f5df0d2920de25364b1b4eb9118243

To re-run the test, copy/paste the following into the shell.
You may also copy/paste (except the first line) into an argfile
8<----------------------------------------
./stester \
\ --rebalance-mode out --workload HybridWorkloadGroup
\ --rebalance-count 2 --testcase RebalanceScenario -C share\rexec
\ --rexec_path C:\temp\sdkd-out-debug\SdkdConsole.exe --rexec_port
\ 8675 --cluster_node=10.3.121.134 --cluster_node=10.3.121.135
\ --cluster_node=10.3.121.136 --cluster_node=10.3.3.206
--cluster_ssh-username=root --cluster_ssh-password=couchbase
---------------------------------------->8
[1,49 WARN] (Drivers getDriver:76) 'rexec' is now mapped to local execution only. Use RemoteExecutingDriver for remote execution
[1,77 INFO] (RunContext run:102) Ramp for 30 seconds. Cluster modification: remove 2 nodes and rebalance. Rebound for 90 seconds.
[1,78 INFO] (RunContext run:124) Starting cluster and driver
[1,78 INFO] (CBCluster startCluster:360) Node http://10.3.121.134:8091 is master now
[1,78 INFO] (HostPortDriver launch:32) Invoking SDKD as 'C:\temp\sdkd-out-debug\SdkdConsole.exe'
[1,78 DEBUG] (CBCluster startCluster:366) Stopping any existing rebalance operations..
[1,97 INFO] (SDKD log:137) — Logging Self-Test —
[1,98 INFO] (SDKD log:137) [Sdkd.Main|Info] Info Message
[1,98 INFO] (SDKD log:137) [Sdkd.Main|Warn] Warn Message
[1,98 INFO] (SDKD log:137) [Sdkd.Main|Error] Error Message
[1,98 INFO] (SDKD log:137) [Sdkd.Main|Fatal] Fatal Message
[1,98 INFO] (SDKD log:137) [Sdkd.Main|Info] SDKD Listening on port 8675
[2,33 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[2,33 DEBUG] (Handle sendMessageAsync:183) > INFO@0.0
[2,51 DEBUG] (Handle receiveMessage:158) < INFO@0.0 => {"CAPS":

{"DS_SHARED":true,"PREAMBLE":false,"CANCEL":true,"CONTINUOUS":true}

,"COMPONENTS":{"SDK":"1.0.0.0","CLR":"4.0.30319.34014"}}
[8,05 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.135:8091,ns_1@10.3.121.135>
[8,55 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.121.136:8091,ns_1@10.3.121.136>
[9,05 DEBUG] (CBCluster clearSingleCluster:140) Failing over existing node <URI:10.3.3.206:8091,ns_1@10.3.3.206>
[9,56 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[9,57 DEBUG] (SSHConnection connect:99) Connecting with User[root] Pass[HASH=3b8cf05627127baaf3808913209aac84]
[11,58 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.136:8091
[11,59 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.135:8091
[11,59 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.121.134:8091
[11,59 INFO] (NodeHost createSSH:147) SSH Initialized for http://10.3.3.206:8091
[11,61 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.3.206
[11,61 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.134
[11,61 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.136
[11,61 DEBUG] (SSHCommand execute:75) Running /etc/init.d/couchbase-server start && pkill -CONT -f memcached && pkill -CONT -f beam.smp && iptables -F && iptables -t nat -F on 10.3.121.135
[12,45 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@1bd5211
[12,45 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@3456dafb
[12,46 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@7bac9f86
[12,46 DEBUG] (SSHCommand close:147) Closing channel com.jcraft.jsch.ChannelExec@76be3008
[12,46 DEBUG] (CBCluster setupNewCluster:271) Provisioning initial node com.couchbase.cbadmin.client.CouchbaseAdmin@53ac4ca1
[28,46 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.135:8091
[33,13 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.121.136:8091
[43,50 DEBUG] (CBCluster tryOnce:286) Adding node http://10.3.3.206:8091
[51,51 INFO] (CBCluster setupNewCluster:293) All nodes added. Will rebalance
[52,51 INFO] (RebalanceWaiter sweepOnce:33) Rebalance complete
[52,51 DEBUG] (CBCluster setupServerGroups:222) Not creating any groups
[52,76 INFO] (CBCluster setupMainBucket:209) Creating bucket default
[53,01 INFO] (CBCluster setupMainBucket:211) Bucket creation submitted
[60,67 INFO] (CBCluster waitForBucketReady:203) Bucket creation done
[63,56 INFO] (RunContext run:143) Driver and cluster initialized
[63,82 INFO] (RunContext call:167) Running scenario..
[63,82 INFO] (Scenario run:72) Starting RAMP phase
[63,83 INFO] (Workload setupDesign:63) Creating design test_design
[65,11 INFO] (Workload setupDesign:80) Design creation done
[65,12 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[65,12 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@101.1 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[65,13 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 101
[65,14 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Creating new shared client for key 'U,P,H10.3.121.134:8091,Bdefault'
[65,16 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving Common.Logging.Log4Net
[65,16 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly Common.Logging.Log4Net, Version=2.0.0.0, Culture=neutral, PublicKeyToken=af08829b84f0328e
[65,18 INFO] (SDKD log:137) [Sdkd.Main|Info] Resolving log4net
[65,19 INFO] (SDKD log:137) [Sdkd.Main|Info] Have assembly log4net, Version=1.2.10.0, Culture=neutral, PublicKeyToken=1b44e1d426115821
[69,26 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@101.1
[69,26 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@101.2 => {DSType=DSTYPE_SEEDED, DS={KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}, Schema={InflateContent=meh, InflateLevel=40}, O
ptions={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[76,93 INFO] (SDKD log:137) [Sdkd.Result|Warn] Unrecognized return code 409
[77,18 INFO] (SDKD log:137) [Sdkd.ViewLoader|Warn] there are no OK responses at ViewLoader
[77,18 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {
[77,18 INFO] (SDKD log:137) "0": 20,
[77,18 INFO] (SDKD log:137) "264": 980
[77,19 DEBUG] (Handle receiveMessage:158) < CB_VIEW_LOAD@101.2 => {[OK]: 20, [MEMD:GENERIC]: 980}
[77,19 INFO] (SDKD log:137) }
[77,19 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@101.3
[77,20 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 269687548
[77,20 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@102.4 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}

, Hostname=10.3.121.134}
[77,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[77,20 INFO] (SDKD log:137) dispose U,P,H10.3.121.134:8091,Bdefault CouchbaseBucket connection from sdkd
[77,20 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 102
[77,21 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Creating new shared client for key 'U,P,H10.3.121.134:8091,Bdefault'
[81,00 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@102.4
[81,01 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_LOAD@102.5 => {DSType=DSTYPE_SEEDED, DS=

{KSeed=ViewFillerSeed, VSeed=ViewFillerSeed, Count=1000, Continuous=false, VSize=512, Repeat=rep, KSize=12}

, Schema=

{InflateContent=meh, InflateLevel=40}

, O
ptions={DelayMax=0, DelayMin=0, IterWait=1, TimeRes=0}}
[92,70 INFO] (SDKD log:137) [Sdkd.ViewLoader|Warn] there are no OK responses at ViewLoader
[92,70 INFO] (SDKD log:137) [Sdkd.ViewLoader|Info] Summary details: {
[92,70 DEBUG] (Handle receiveMessage:158) < CB_VIEW_LOAD@102.5 =>

{[OK]: 36, [MEMD:GENERIC]: 964}

[92,70 INFO] (SDKD log:137) "0": 36,
[92,70 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@102.6
[92,70 INFO] (SDKD log:137) "264": 964
[92,71 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@103.7 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[92,71 INFO] (SDKD log:137) }
[92,71 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Release CouchbaseBucket 104406921
[92,71 INFO] (SDKD log:137) dispose U,P,H10.3.121.134:8091,Bdefault CouchbaseBucket connection from sdkd
[92,72 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[92,72 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 103
[92,72 INFO] (SDKD log:137) [Sdkd.ClientFactory|Info] Creating new shared client for key 'U,P,H10.3.121.134:8091,Bdefault'
[96,62 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@103.7
[96,62 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@103.8 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=test
_view}
[96,63 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[96,63 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@104.9 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}

, Hostname=10.3.121.134}
[96,64 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 104
[96,64 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@104.9
[96,64 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@104.10 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options=

{DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=tes
t_view}
[96,65 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[96,65 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@105.11 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[96,66 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 105
[96,66 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@105.11
[96,67 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@105.12 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}

, ViewName=tes
t_view}
[96,68 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[96,68 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@106.13 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}, Hostname=10.3.121.134}
[96,68 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 106
[96,69 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@106.13
[96,69 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@106.14 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options={DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}, ViewName=tes
t_view}
[96,69 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[96,69 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@107.15 => {Port=8091, Bucket=default, Options={SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}

, Hostname=10.3.121.134}
[96,70 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 107
[96,70 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@107.15
[96,70 DEBUG] (Handle sendMessageAsync:183) > CB_VIEW_QUERY@107.16 => {DSType=DSTYPE_SEEDED, DS={}, ViewParameters={}, DesignName=test_design, Options=

{DelayMax=10, DelayMin=1, IterWait=1, ViewQueryCount=-1, ViewQueryDelay=0, TimeRes=1}

, ViewName=tes
t_view}
[96,71 INFO] (Workload call:112) Preloading items into cluster before workload
[96,71 INFO] (SDKD log:137) [Sdkd.Control|Info] Got a new connection. Creating child handle
[96,72 DEBUG] (Handle sendMessageAsync:183) > NEWHANDLE@108.17 => {Port=8091, Bucket=default, Options=

{SSL=false, Username=default, Password=, OtherNodes=[["10.3.121.135",8091],["10.3.121.136",8091],["10.3.3.206",8091]]}

, Hostname=10.3.121.134}
[96,72 INFO] (SDKD log:137) [Sdkd.Control|Info] Registering handle 108
[96,72 DEBUG] (Handle receiveMessage:158) < NEWHANDLE@108.17
[96,72 DEBUG] (Handle sendMessageAsync:183) > MC_DS_MUTATE_SET@108.18 => {DSType=DSTYPE_SEEDED, DS=

{KSeed=SimpleKey, VSeed=SimpleValue, Count=15000, Continuous=false, VSize=128, Repeat=REP, KSize=32}

, Options={DelayMax=0, DelayMin=0, IterWait=100, Ti
meRes=0}}
[99,64 INFO] (SDKD log:137)
[99,64 INFO] (SDKD log:137) Unhandled Exception: System.InvalidOperationException: Collection was modified; enumeration operation may not execute
[99,65 INFO] (SDKD log:137) at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
[99,65 INFO] (SDKD log:137) at System.Collections.Generic.List`1.ForEach(Action`1 action)
[99,65 INFO] (SDKD log:137) at Couchbase.Configuration.ConfigContextBase.Dispose(Boolean disposing) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 197
[99,65 INFO] (SDKD log:137) at Couchbase.Configuration.ConfigContextBase.Dispose() at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\ConfigContextBase.cs:line 181
[99,66 INFO] (SDKD log:137) at Couchbase.Configuration.Server.Providers.CarrierPublication.CarrierPublicationProvider.UnRegisterObserver(IConfigObserver observer) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Configuration\Server\P
roviders\CarrierPublication\CarrierPublicationProvider.cs:line 131
[99,66 INFO] (SDKD log:137) at Couchbase.Core.ClusterManager.DestroyBucket(IBucket bucket) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 228
[99,67 INFO] (SDKD log:137) at Couchbase.Core.ClusterManager.Dispose(Boolean disposing) at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 250
[99,67 INFO] (SDKD log:137) at Couchbase.Core.ClusterManager.Dispose() at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\Core\ClusterManager.cs:line 235
[99,67 INFO] (SDKD log:137) at Couchbase.CouchbaseCluster.Finalize() at c:\jenkins\workspace-net2\couchbase-net-client\Src\Couchbase\CouchbaseCluster.cs:line 165
[115,39 ERROR] (BRun maybePropagate:237) Suppressing driver-related error
com.couchbase.sdkdclient.context.HarnessException: Driver crashed
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:192) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.context.RunContext$5.call(RunContext.java:183) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:1.7.0_60]
at java.lang.Thread.run(Unknown Source) ~[na:1.7.0_60]
[115,42 DEBUG] (Handle sendMessageAsync:183) > CLOSEHANDLE@103.19
[115,43 WARN] (STester close:170) While stopping driver
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.7.0_60]
at java.net.SocketOutputStream.socketWrite(Unknown Source) ~[na:1.7.0_60]
at java.net.SocketOutputStream.write(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source) ~[na:1.7.0_60]
at sun.nio.cs.StreamEncoder.flush(Unknown Source) ~[na:1.7.0_60]
at java.io.OutputStreamWriter.flush(Unknown Source) ~[na:1.7.0_60]
at java.io.BufferedWriter.flush(Unknown Source) ~[na:1.7.0_60]
at com.couchbase.sdkdclient.handle.Handle.sendMessageAsync(Handle.java:188) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.handle.Handle.close(Handle.java:254) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HandleManager.stopAll(HandleManager.java:26) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.HostPortDriver.stop(HostPortDriver.java:118) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.driver.ExecutingDriver.stop(ExecutingDriver.java:117) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.stester.STester.close(STester.java:167) ~[sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runHarness(BRun.java:281) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.runSingleTest(BRun.java:388) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.run(BRun.java:472) [sdkdclient-1.0-SNAPSHOT.jar:na]
at com.couchbase.sdkdclient.batch.BRun.main(BRun.java:504) [sdkdclient-1.0-SNAPSHOT.jar:na]