deeptida@ubu-1512:~/testing/cbsdkd$ ./stester -i /home/deeptida/testing/mark-vm.ini -c failover.ReAdd --dsw_timeres 1 --cl_bucket_type memcached --cl_bucket_password verysecret --cl_bucket_name a_memcached_bucket -C share/rexec --rexec_path /home/deeptida/testing/sdkd-java/run-sdkd-java --rexec_port 8050 --num_nodes 1 --action_delay 30 -o mem_bkt.log Established testrunner/rc logging ./lib/cbsdk/plumber.py:63: UserWarning: ConfigOptionCollection is deprecated ret = Driver2Config.load(s) ./stester:188: UserWarning: -i/--ini is deprecated. This will override any --nodes_* arguments warn("-i/--ini is deprecated. This will override any --nodes_* arguments") action => 'FO_NOACTION' action_delay => 30 cl_bucket_add_dfl => True cl_bucket_name => 'a_memcached_bucket' cl_bucket_password => 'verysecret' cl_bucket_ram => 512 cl_bucket_replicas => 1 cl_bucket_type => 'memcached' cl_fake => False cl_nodes_ok => False cl_noinit => False cl_replicas => 1 cl_use_backups => False dsw_continuous => True dsw_delay => -1 dsw_delay_max => 10 dsw_delay_min => 1 dsw_iterwait => 1 dsw_kseed => 'SimpleKey_' dsw_ksize => 12 dsw_kvcount => 15000 dsw_nthreads => 10 dsw_opmode => 'both' dsw_timeres => 1 dsw_ttl => 0 dsw_vsize => 100 ept => False grace_after => 60 no_fo => False node => ['10.3.121.207', '10.3.121.209', '10.3.121.208', '10.3.121.212'] nodes_rest_password => '123456' nodes_rest_port => 8091 nodes_rest_user => 'Administrator' nodes_ssh_disable => False nodes_ssh_key => None nodes_ssh_password => 'root' nodes_ssh_user => 'root' num_nodes => 1 portaction => None portservice => None ramp => 5 readd_rb => 'RB_AFTER' readd_wait => 30 rebound => 90 rexec_password => None rexec_path => '/home/deeptida/testing/sdkd-java/run-sdkd-java' rexec_port => 8050 rexec_wait => 2 service => None svc_itergrace => 0 svc_itergrace_off => 0 svc_itergrace_on => 0 svcaction => 'RESTART' --- Initializing SDKD --- --- Initializing Global Settings --- --- Running --- [INFO 0.23 cbsdk.cluster.node node.py:362] Unblocking firewall for nodes [INFO 0.99 cbsdk.cluster.node node.py:182] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 0.99 cbsdk.cluster.node node.py:182] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 1.15 cbsdk.cluster.node node.py:149] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.18 cbsdk.cluster.node node.py:149] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.22 cbsdk.cluster.node node.py:182] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 1.30 cbsdk.cluster.ssh ssh.py:108] outbuf closed [INFO 1.68 cbsdk.cluster.node node.py:149] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.94 cbsdk.cluster.ssh ssh.py:108] outbuf closed Exception in thread Thread-8: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "/usr/lib/python2.6/threading.py", line 484, in run self.__target(*self.__args, **self.__kwargs) File "./lib/cbsdk/cluster/ssh.py", line 250, in connect self._ssh.connect(**pargs) File "/usr/lib/pymodules/python2.6/paramiko/client.py", line 327, in connect self._auth(username, password, pkey, key_filenames, allow_agent, look_for_keys) File "/usr/lib/pymodules/python2.6/paramiko/client.py", line 481, in _auth raise saved_exception AuthenticationException: Authentication failed. [WARNING 2.30 paramiko.transport transport.py:1386] Oops, unhandled type 3 Exception in thread Thread-25: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "./lib/cbsdk/cluster/ssh.py", line 127, in run chan = self.sshclient.get_transport().open_session() File "/usr/lib/pymodules/python2.6/paramiko/transport.py", line 655, in open_session return self.open_channel('session') File "/usr/lib/pymodules/python2.6/paramiko/transport.py", line 745, in open_channel raise e EOFError [INFO 119.87 cbsdk.cluster.node node.py:182] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] Exception in thread Thread-26: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "./lib/cbsdk/cluster/ssh.py", line 130, in run stdin = chan.makefile('wb') AttributeError: 'NoneType' object has no attribute 'makefile' Exception in thread Thread-27: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "./lib/cbsdk/cluster/ssh.py", line 130, in run stdin = chan.makefile('wb') AttributeError: 'NoneType' object has no attribute 'makefile' [INFO 119.88 cbsdk.cluster.node node.py:149] --START: ['/etc/init.d/couchbase-server start'] Exception in thread Thread-28: Traceback (most recent call last): File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner self.run() File "./lib/cbsdk/cluster/ssh.py", line 130, in run stdin = chan.makefile('wb') AttributeError: 'NoneType' object has no attribute 'makefile' [INFO 124.51 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.212 ssh_username:root is apparently uninitialized [INFO 124.52 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.208 ssh_username:root is apparently uninitialized [INFO 124.53 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.209 ssh_username:root is apparently uninitialized [INFO 124.53 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.207 ssh_username:root is apparently uninitialized [INFO 124.56 cbsdk.cluster.cluster cluster.py:360] Using ip:10.3.121.212 ssh_username:root as master [INFO 124.89 cbsdk.tr-rc rest_client.py:670] adding remote node : 10.3.121.208 to this cluster @ : 10.3.121.212 [INFO 125.87 cbsdk.tr-rc rest_client.py:670] adding remote node : 10.3.121.209 to this cluster @ : 10.3.121.212 [INFO 126.35 cbsdk.tr-rc rest_client.py:670] adding remote node : 10.3.121.207 to this cluster @ : 10.3.121.212 [INFO 126.87 cbsdk.tr-rc rest_client.py:749] rebalance params : password=123456&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.121.212%2Cns_1%4010.3.121.208%2Cns_1%4010.3.121.209%2Cns_1%4010.3.121.207 [INFO 126.94 cbsdk.tr-rc rest_client.py:756] rebalance operation started [INFO 126.95 cbsdk.tr-rc rest_client.py:824] rebalance percentage : 0 % [INFO 136.97 cbsdk.cluster.cluster cluster.py:375] Rebalance done.. [INFO 137.01 cbsdk.tr-rc rest_client.py:1107] http://10.3.121.212:8091//pools/default/buckets with param: proxyPort=11211&bucketType=memcached&authType=sasl&replicaIndex=0&name=a_memcached_bucket&saslPassword=verysecret&replicaNumber=1&ramQuotaMB=512 [INFO 137.07 cbsdk.tr-rc rest_client.py:1112] 0.0621891021729 seconds to create bucket a_memcached_bucket [INFO 137.07 cbsdk.cluster.cluster cluster.py:89] Not waiting on memcached bucket.. [INFO 137.07 cbsdk.cluster.cluster cluster.py:92] Also creating default bucket... [INFO 137.07 cbsdk.tr-rc rest_client.py:1107] http://10.3.121.212:8091//pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=0&name=default&saslPassword=&replicaNumber=1&ramQuotaMB=256 [INFO 137.13 cbsdk.tr-rc rest_client.py:1112] 0.057158946991 seconds to create bucket default [INFO 139.03 cbsdk.util.mcwait mcwait.py:196] Started thread for ip:10.3.121.212 ssh_username:root [INFO 139.03 cbsdk.util.mcwait mcwait.py:196] Started thread for ip:10.3.121.207 ssh_username:root [INFO 139.03 cbsdk.util.mcwait mcwait.py:196] Started thread for ip:10.3.121.209 ssh_username:root [INFO 139.04 cbsdk.util.mcwait mcwait.py:196] Started thread for ip:10.3.121.208 ssh_username:root [INFO 145.47 cbsdk.cluster.cluster cluster.py:394] Using cluster version 2.0.1-173-rel-enterprise [SDKD(WARNING) 145.48 cbsdk.sdkd.local executor.py:66] + pgrep -f java.+-jar.*sdkd-java [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + dirname /home/deeptida/testing/sdkd-java/run-sdkd-java [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + SDKD_ROOT=/home/deeptida/testing/sdkd-java [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + cd /home/deeptida/testing/sdkd-java [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + JAR= [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + JAR_DEFAULT= [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + [ -z ] [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + JAR_DEFAULT=1 [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + JAR=/home/deeptida/testing/sdkd_exec.jar [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + [ ! -e /home/deeptida/testing/sdkd_exec.jar ] [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + [ ! -e /home/deeptida/testing/sdkd_exec.jar ] [SDKD(WARNING) 145.49 cbsdk.sdkd.local executor.py:66] + java -ea -jar /home/deeptida/testing/sdkd_exec.jar -l 8050 -shared 100 -sync [SDKD(WARNING) 147.66 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 147.66 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 147.67 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 147.67 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 147.86 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM com.couchbase.client.CouchbaseProperties setPropertyFile [SDKD(WARNING) 147.86 cbsdk.sdkd.local executor.py:66] INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader. [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.212:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.209:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.208:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 147.89 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 147.91 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 147.91 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@571a75a2 [SDKD(WARNING) 147.91 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM com.couchbase.client.CouchbaseClient [SDKD(WARNING) 147.91 cbsdk.sdkd.local executor.py:66] INFO: viewmode property isn't defined. Setting viewmode to production mode [SDKD(WARNING) 147.92 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 147.92 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@351563ff [SDKD(WARNING) 147.93 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 147.93 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@a210b5b [SDKD(WARNING) 147.93 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 147.93 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@7f70888e [SDKD(WARNING) 148.07 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.auth.AuthThread$1 receivedStatus [SDKD(WARNING) 148.07 cbsdk.sdkd.local executor.py:66] INFO: Authenticated to /10.3.121.212:11210 [SDKD(WARNING) 148.07 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.auth.AuthThread$1 receivedStatus [SDKD(WARNING) 148.07 cbsdk.sdkd.local executor.py:66] INFO: Authenticated to /10.3.121.209:11210 [SDKD(WARNING) 148.07 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.auth.AuthThread$1 receivedStatus [SDKD(WARNING) 148.07 cbsdk.sdkd.local executor.py:66] INFO: Authenticated to /10.3.121.207:11210 [SDKD(WARNING) 148.08 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:39 AM net.spy.memcached.auth.AuthThread$1 receivedStatus [SDKD(WARNING) 148.08 cbsdk.sdkd.local executor.py:66] INFO: Authenticated to /10.3.121.208:11210 [SDKD(WARNING) 148.24 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.24 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.24 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.24 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.25 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.25 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.25 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.25 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.26 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.26 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.26 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.26 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.27 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.28 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.29 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.30 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.30 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.30 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.30 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.31 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.32 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.32 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.32 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.32 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.32 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.33 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.33 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.33 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.33 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.33 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.33 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.34 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.34 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.34 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.34 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.34 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.35 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.35 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.35 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.35 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:40 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 148.36 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [INFO 148.41 cbsdk.scenario base.py:51] Ramp-up for 5 seconds [PHASE 148.41 cbsdk.scenario logger.py:272] Changing phase to: RAMP [PHASE 153.42 cbsdk.scenario logger.py:275] Changing phase to: CHANGE [INFO 153.42 cbsdk.scenario failover.py:274] No services specified [ACTION 153.42 cbsdk.scenario failover.py:276] Failing over [ip:10.3.121.207 ssh_username:root] [ACTION 153.42 cbsdk.cluster.cluster cluster.py:700] Failing over node ip:10.3.121.207 ssh_username:root [INFO 153.67 cbsdk.tr-rc rest_client.py:733] fail_over successful [ACTION 153.67 cbsdk.cluster.cluster cluster.py:727] No post-failover action requested [ACTION 153.67 cbsdk.scenario failover.py:281] Sleeping for another 60 seconds [SDKD(WARNING) 153.68 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 153.68 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to exception on {QA sa=/10.3.121.207:11210, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 11301 Key: SimpleKey__REP_542 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, topWop=null, toWrite=0, interested=1} [SDKD(WARNING) 153.68 cbsdk.sdkd.local executor.py:66] java.io.IOException: Disconnected unexpected, will reconnect. [SDKD(WARNING) 153.68 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:526) [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:454) [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=2, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 11301 Key: SimpleKey__REP_542 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, topWop=null, toWrite=0, interested=1}, attempt 0. [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 11301 Key: SimpleKey__REP_542 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 11311 Key: SimpleKey__REP_552 [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout [SDKD(WARNING) 153.69 cbsdk.sdkd.local executor.py:66] WARNING: Unknown exception encountered (for operation) future warnings will be suppressed [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] java.util.concurrent.CancellationException: Cancelled [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:170) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:114) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:131) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:68) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 153.70 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] WARNING: Unknown exception encountered (for operation) future warnings will be suppressed [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] java.util.concurrent.CancellationException: Cancelled [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:170) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:114) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:131) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:64) [SDKD(WARNING) 153.71 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 153.72 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 153.73 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 153.73 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 153.73 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 153.73 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:25:45 AM com.couchbase.client.CouchbaseMemcachedConnection reconfigure [SDKD(WARNING) 153.73 cbsdk.sdkd.local executor.py:66] INFO: Scheduling Node /10.3.121.207:11210for shutdown. [ACTION 213.74 cbsdk.cluster.cluster cluster.py:758] Re-Added ip:10.3.121.207 ssh_username:root (True) [INFO 213.74 cbsdk.cluster.cluster cluster.py:772] to_add: [ip:10.3.121.212 ssh_username:root, ip:10.3.121.208 ssh_username:root, ip:10.3.121.207 ssh_username:root, ip:10.3.121.209 ssh_username:root] [ACTION 213.74 cbsdk.scenario failover.py:307] Rebalancing after re-add [INFO 214.11 cbsdk.tr-rc rest_client.py:749] rebalance params : password=123456&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.121.212%2Cns_1%4010.3.121.208%2Cns_1%4010.3.121.209%2Cns_1%4010.3.121.207 [INFO 214.17 cbsdk.tr-rc rest_client.py:756] rebalance operation started [INFO 214.18 cbsdk.tr-rc rest_client.py:824] rebalance percentage : 0 % [INFO 224.20 cbsdk.tr-rc rest_client.py:824] rebalance percentage : 15.8573706857 % [INFO 234.23 cbsdk.tr-rc rest_client.py:824] rebalance percentage : 35.5475238787 % [INFO 244.26 cbsdk.tr-rc rest_client.py:824] rebalance percentage : 50.0 % [SDKD(WARNING) 244.35 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:16 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 244.35 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 244.35 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:16 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 244.35 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=8}, attempt 1. [SDKD(WARNING) 245.09 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:16 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 245.09 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=20, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 305133 Key: SimpleKey__REP_15475 Cas: 0 Exp: 0 Flags: 0 Data Length: 102, toWrite=0, interested=0}, attempt 2. [SDKD(WARNING) 245.09 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:16 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 245.09 cbsdk.sdkd.local executor.py:66] INFO: The channel or socket was null for {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=20, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 305133 Key: SimpleKey__REP_15475 Cas: 0 Exp: 0 Flags: 0 Data Length: 102, toWrite=0, interested=0} [SDKD(WARNING) 248.36 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:20 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 248.36 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=40, #iq=0, topRop=null, topWop=Cmd: 1 Opaque: 305133 Key: SimpleKey__REP_15475 Cas: 0 Exp: 0 Flags: 0 Data Length: 102, toWrite=0, interested=0} [SDKD(WARNING) 248.36 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:20 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 248.36 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@1838de99 [SDKD(WARNING) 248.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:20 AM net.spy.memcached.auth.AuthThread$1 receivedStatus [SDKD(WARNING) 248.37 cbsdk.sdkd.local executor.py:66] INFO: Authenticated to /10.3.121.207:11210 [SDKD(WARNING) 253.09 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:27:24 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 253.09 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=5, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 253.09 cbsdk.sdkd.local executor.py:66] Exception in thread "Memcached IO over {MemcachedConnection to /10.3.121.212:11210 /10.3.121.209:11210 /10.3.121.208:11210 /10.3.121.207:11210}" java.lang.AssertionError: Attempting to overwrite channel [SDKD(WARNING) 253.09 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.protocol.TCPMemcachedNodeImpl.setChannel(TCPMemcachedNodeImpl.java:496) [SDKD(WARNING) 253.09 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.protocol.TCPMemcachedNodeImpl.registerChannel(TCPMemcachedNodeImpl.java:484) [SDKD(WARNING) 253.09 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.attemptReconnects(MemcachedConnection.java:680) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-19: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-6: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 253.10 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-10: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-5: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.11 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-20: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 253.12 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-8: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-11: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 253.13 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [ACTION 254.27 cbsdk.scenario failover.py:311] Rebalance done... [PHASE 254.28 cbsdk.scenario logger.py:278] Changing phase to: REBOUND [ACTION 254.28 cbsdk.scenario base.py:59] Sleeping 90 seconds before collection [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-17: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-16: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.59 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-9: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-3: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-1: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.60 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-13: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-2: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.61 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-7: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-12: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-15: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.62 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-14: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-4: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.63 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-18: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 255.64 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [PHASE 344.37 cbsdk.scenario logger.py:281] Changing phase to: DONE [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.37 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 344.38 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. ./stester:235: UserWarning: Got exception running the test warn("Got exception running the test") [ERROR 344.42 cbsdk.__main__ stester:236] Couldn't decode (No JSON object could be decoded) Traceback (most recent call last): File "./stester", line 226, in run_test obj.run() File "/home/deeptida/testing/cbsdkd/lib/cbsdk/scenarios/failover.py", line 314, in run workload.collect() File "/home/deeptida/testing/cbsdkd/lib/cbsdk/workloads/getset.py", line 228, in collect resp = h.conduit.recv_msg() File "./lib/cbsdk/driver.py", line 47, in recv_msg msg = Res.Response.parse(txt) File "./lib/cbsdk/response.py", line 147, in parse txt, e)) ProtocolError: Couldn't decode (No JSON object could be decoded) --- Done --- [SDKD(WARNING) 344.43 cbsdk.sdkd.local executor.py:66] Mar 12, 2013 6:28:56 AM com.couchbase.sdkd.server.ControlServer run Output now empty. Breaking