deeptida@ubu-1512:~/testing/cbsdkd$ ./stester -i /home/deeptida/testing/mark-vm.ini -c failover.Once --no_fo 1 --service ALL --svcaction STOP --grace_after 60 --dsw_timeres 1 -d -o restart_memcached_cluster.log --cl_bucket_type memcached -C share/rexec --rexec_path /home/deeptida/testing/sdkd-java/run-sdkd-java --rexec_port 8050 --dsw_delay_min 0 --dsw_delay_max 0 --num_nodes 1 Established testrunner/rc logging ./lib/cbsdk/plumber.py:63: UserWarning: ConfigOptionCollection is deprecated ret = Driver2Config.load(s) ./stester:188: UserWarning: -i/--ini is deprecated. This will override any --nodes_* arguments warn("-i/--ini is deprecated. This will override any --nodes_* arguments") action => 'FO_NOACTION' action_delay => 0 cl_bucket_add_dfl => True cl_bucket_name => 'default' cl_bucket_password => None cl_bucket_ram => 512 cl_bucket_replicas => 1 cl_bucket_type => 'memcached' cl_fake => False cl_nodes_ok => False cl_noinit => False cl_replicas => 1 cl_use_backups => False dsw_continuous => True dsw_delay => -1 dsw_delay_max => 0 dsw_delay_min => 0 dsw_iterwait => 1 dsw_kseed => 'SimpleKey_' dsw_ksize => 12 dsw_kvcount => 1000 dsw_nthreads => 10 dsw_opmode => 'both' dsw_timeres => 1 dsw_ttl => 0 dsw_vsize => 100 ept => False grace_after => 60 no_fo => True node => ['10.3.121.207', '10.3.121.209', '10.3.121.208', '10.3.121.212'] nodes_rest_password => '123456' nodes_rest_port => 8091 nodes_rest_user => 'Administrator' nodes_ssh_disable => False nodes_ssh_key => None nodes_ssh_password => 'root' nodes_ssh_user => 'root' num_nodes => 1 portaction => None portservice => None ramp => 5 rebound => 90 rexec_password => None rexec_path => '/home/deeptida/testing/sdkd-java/run-sdkd-java' rexec_port => 8050 service => 'ALL' svc_itergrace => 0 svc_itergrace_off => 0 svc_itergrace_on => 0 svcaction => 'STOP' --- Initializing SDKD --- --- Initializing Global Settings --- --- Running --- [INFO 0.20 cbsdk.cluster.node node.py:359] Unblocking firewall for nodes [INFO 0.86 cbsdk.cluster.node node.py:179] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 0.97 cbsdk.cluster.node node.py:179] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 0.99 cbsdk.cluster.node node.py:179] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 1.02 cbsdk.cluster.node node.py:146] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.14 cbsdk.cluster.node node.py:146] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.14 cbsdk.cluster.node node.py:179] --RESUME: ['killall -CONT memcached', 'killall -CONT beam.smp'] [INFO 1.18 cbsdk.cluster.ssh ssh.py:108] outbuf closed [INFO 1.22 cbsdk.cluster.node node.py:146] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.39 cbsdk.cluster.ssh ssh.py:108] outbuf closed [INFO 1.55 cbsdk.cluster.node node.py:146] --START: ['/etc/init.d/couchbase-server start'] [INFO 1.80 cbsdk.cluster.ssh ssh.py:108] outbuf closed [DEBUG 1.82 cbsdk.cluster.cluster cluster.py:262] Successfully stopped rebalance on ip:10.3.121.212 ssh_username:root [DEBUG 1.83 cbsdk.cluster.cluster cluster.py:262] Successfully stopped rebalance on ip:10.3.121.208 ssh_username:root [DEBUG 1.83 cbsdk.cluster.cluster cluster.py:262] Successfully stopped rebalance on ip:10.3.121.209 ssh_username:root [DEBUG 1.84 cbsdk.cluster.cluster cluster.py:262] Successfully stopped rebalance on ip:10.3.121.207 ssh_username:root [DEBUG 1.88 cbsdk.cluster.cluster cluster.py:275] Found buckets [] [DEBUG 1.89 cbsdk.cluster.cluster cluster.py:275] Found buckets [] [DEBUG 1.89 cbsdk.cluster.cluster cluster.py:275] Found buckets [] [DEBUG 1.90 cbsdk.cluster.cluster cluster.py:275] Found buckets [] [INFO 1.94 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.212 ssh_username:root is apparently uninitialized [INFO 1.95 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.208 ssh_username:root is apparently uninitialized [INFO 1.95 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.209 ssh_username:root is apparently uninitialized [INFO 1.96 cbsdk.cluster.cluster cluster.py:289] Node ip:10.3.121.207 ssh_username:root is apparently uninitialized [INFO 1.98 cbsdk.cluster.cluster cluster.py:360] Using ip:10.3.121.212 ssh_username:root as master [INFO 2.81 cbsdk.tr-rc rest_client.py:670] adding remote node : 10.3.121.208 to this cluster @ : 10.3.121.212 [INFO 4.16 cbsdk.tr-rc rest_client.py:670] adding remote node : 10.3.121.209 to this cluster @ : 10.3.121.212 [INFO 5.44 cbsdk.tr-rc rest_client.py:670] adding remote node : 10.3.121.207 to this cluster @ : 10.3.121.212 [INFO 6.59 cbsdk.tr-rc rest_client.py:749] rebalance params : password=123456&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.121.212%2Cns_1%4010.3.121.208%2Cns_1%4010.3.121.209%2Cns_1%4010.3.121.207 [INFO 6.65 cbsdk.tr-rc rest_client.py:756] rebalance operation started [INFO 6.67 cbsdk.tr-rc rest_client.py:824] rebalance percentage : 0 % [INFO 16.69 cbsdk.cluster.cluster cluster.py:375] Rebalance done.. [INFO 16.69 cbsdk.tr-rc rest_client.py:1107] http://10.3.121.212:8091//pools/default/buckets with param: proxyPort=11211&bucketType=memcached&authType=sasl&replicaIndex=0&name=default&saslPassword=&replicaNumber=1&ramQuotaMB=512 [INFO 16.76 cbsdk.tr-rc rest_client.py:1112] 0.0708241462708 seconds to create bucket default [INFO 16.76 cbsdk.cluster.cluster cluster.py:89] Not waiting on memcached bucket.. [INFO 16.89 cbsdk.cluster.cluster cluster.py:394] Using cluster version 2.0.0-1976-rel-enterprise [DEBUG 16.89 cbsdk.sdkdfactory.executor executor.py:58] Spawning: ['/home/deeptida/testing/sdkd-java/run-sdkd-java'] [SDKD(WARNING) 16.89 cbsdk.sdkd.local executor.py:66] + pgrep -f java.+-jar.*sdkd-java [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + dirname /home/deeptida/testing/sdkd-java/run-sdkd-java [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + SDKD_ROOT=/home/deeptida/testing/sdkd-java [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + cd /home/deeptida/testing/sdkd-java [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + JAR= [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + JAR_DEFAULT= [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + [ -z ] [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + JAR_DEFAULT=1 [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + JAR=/home/deeptida/testing/sdkd_exec.jar [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + [ ! -e /home/deeptida/testing/sdkd_exec.jar ] [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + [ ! -e /home/deeptida/testing/sdkd_exec.jar ] [SDKD(WARNING) 16.90 cbsdk.sdkd.local executor.py:66] + java -ea -jar /home/deeptida/testing/sdkd_exec.jar -l 8050 -shared 100 -sync [DEBUG 18.90 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 18.90 cbsdk.driver driver.py:39] INFO:0 @Control {} [DEBUG 19.01 cbsdk.driver driver.py:48] INFO:0 @0 (OK) {'COMPONENTS': {'SPY': 'Spymemcached 2.8.10\n\nTree Version: 2.8.10-5-g84ebf9a\nLast Commit ID: 84ebf9ad0d0d8a64497d956f6e75bf74cd7883de\n\nCompiled by deeptida@ubu-1512 on Thu Feb 21 02:27:18 PST 2013', 'SDK': 'Couchbase Java Client 1.1.1\n\nTree Version: 1.1.1-10-g2439b6a\nLast Commit ID: 2439b6a86b6ca08d30cf9c8e0b0cadb3b1c73cc3\n\nCompiled by deeptida@ubu-1512 on Thu Feb 21 02:27:10 PST 2013'}, 'CAPS': {'VIEWS': True, 'CONTINUOUS': True, 'DS_SHARED': True, 'TTL': True, 'CANCEL': True, 'PREAMBLE': False}} [DEBUG 19.01 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.01 cbsdk.driver driver.py:39] NEWHANDLE:1 @1 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.04 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:33 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.04 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.05 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:33 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.05 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [SDKD(WARNING) 19.23 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.client.CouchbaseProperties setPropertyFile [SDKD(WARNING) 19.23 cbsdk.sdkd.local executor.py:66] INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader. [SDKD(WARNING) 19.25 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 19.25 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.212:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 19.25 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 19.25 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.209:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 19.25 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 19.25 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.208:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 19.26 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection createConnections [SDKD(WARNING) 19.26 cbsdk.sdkd.local executor.py:66] INFO: Added {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue [SDKD(WARNING) 19.27 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.client.CouchbaseClient [SDKD(WARNING) 19.27 cbsdk.sdkd.local executor.py:66] INFO: viewmode property isn't defined. Setting viewmode to production mode [SDKD(WARNING) 19.27 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 19.28 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@571a75a2 [SDKD(WARNING) 19.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 19.30 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@a210b5b [SDKD(WARNING) 19.31 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 19.31 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@7f70888e [SDKD(WARNING) 19.31 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 19.31 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@2c19e15c [DEBUG 19.47 cbsdk.driver driver.py:48] NEWHANDLE:1 @1 (OK) {} [DEBUG 19.47 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.47 cbsdk.driver driver.py:39] NEWHANDLE:2 @2 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.48 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.48 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.48 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.48 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.48 cbsdk.driver driver.py:48] NEWHANDLE:2 @2 (OK) {} [DEBUG 19.48 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.48 cbsdk.driver driver.py:39] NEWHANDLE:3 @3 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.48 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.48 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.49 cbsdk.driver driver.py:48] NEWHANDLE:3 @3 (OK) {} [DEBUG 19.49 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.49 cbsdk.driver driver.py:39] NEWHANDLE:4 @4 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.49 cbsdk.driver driver.py:48] NEWHANDLE:4 @4 (OK) {} [DEBUG 19.49 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.49 cbsdk.driver driver.py:39] NEWHANDLE:5 @5 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.49 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.50 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.50 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.50 cbsdk.driver driver.py:48] NEWHANDLE:5 @5 (OK) {} [DEBUG 19.50 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.50 cbsdk.driver driver.py:39] NEWHANDLE:6 @6 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.50 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.50 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.50 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.50 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.50 cbsdk.driver driver.py:48] NEWHANDLE:6 @6 (OK) {} [DEBUG 19.50 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.50 cbsdk.driver driver.py:39] NEWHANDLE:7 @7 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.51 cbsdk.driver driver.py:48] NEWHANDLE:7 @7 (OK) {} [DEBUG 19.51 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.51 cbsdk.driver driver.py:39] NEWHANDLE:8 @8 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.51 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.51 cbsdk.driver driver.py:48] NEWHANDLE:8 @8 (OK) {} [DEBUG 19.51 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.51 cbsdk.driver driver.py:39] NEWHANDLE:9 @9 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.52 cbsdk.driver driver.py:48] NEWHANDLE:9 @9 (OK) {} [DEBUG 19.52 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.52 cbsdk.driver driver.py:39] NEWHANDLE:10 @10 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.52 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.52 cbsdk.driver driver.py:48] NEWHANDLE:10 @10 (OK) {} [DEBUG 19.52 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.52 cbsdk.driver driver.py:39] NEWHANDLE:11 @11 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.53 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.53 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.53 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.53 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.53 cbsdk.driver driver.py:48] NEWHANDLE:11 @11 (OK) {} [DEBUG 19.53 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.53 cbsdk.driver driver.py:39] NEWHANDLE:12 @12 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.54 cbsdk.driver driver.py:48] NEWHANDLE:12 @12 (OK) {} [DEBUG 19.54 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.54 cbsdk.driver driver.py:39] NEWHANDLE:13 @13 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.54 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.54 cbsdk.driver driver.py:48] NEWHANDLE:13 @13 (OK) {} [DEBUG 19.55 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.55 cbsdk.driver driver.py:39] NEWHANDLE:14 @14 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.55 cbsdk.driver driver.py:48] NEWHANDLE:14 @14 (OK) {} [DEBUG 19.55 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.55 cbsdk.driver driver.py:39] NEWHANDLE:15 @15 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.55 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.56 cbsdk.driver driver.py:48] NEWHANDLE:15 @15 (OK) {} [DEBUG 19.56 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.56 cbsdk.driver driver.py:39] NEWHANDLE:16 @16 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.56 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.56 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.56 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.56 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.56 cbsdk.driver driver.py:48] NEWHANDLE:16 @16 (OK) {} [DEBUG 19.56 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.56 cbsdk.driver driver.py:39] NEWHANDLE:17 @17 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.56 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.56 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.57 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.57 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.57 cbsdk.driver driver.py:48] NEWHANDLE:17 @17 (OK) {} [DEBUG 19.57 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.57 cbsdk.driver driver.py:39] NEWHANDLE:18 @18 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.57 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.57 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.57 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.57 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.57 cbsdk.driver driver.py:48] NEWHANDLE:18 @18 (OK) {} [DEBUG 19.57 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.57 cbsdk.driver driver.py:39] NEWHANDLE:19 @19 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.58 cbsdk.driver driver.py:48] NEWHANDLE:19 @19 (OK) {} [DEBUG 19.58 cbsdk.driver driver.py:350] Created new connection to 8050 [DEBUG 19.58 cbsdk.driver driver.py:39] NEWHANDLE:20 @20 {'Hostname': '10.3.121.212', 'Bucket': 'default', 'Port': 8091, 'Options': {'Username': '', 'Password': '', 'Timeout': None, 'OtherNodes': []}} [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle genHostsList [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] WARNING: URIs: [http://10.3.121.212:8091/pools?#] [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:34 AM com.couchbase.sdkd.cbclient.Handle [SDKD(WARNING) 19.58 cbsdk.sdkd.local executor.py:66] WARNING: JCBC-168: Manually setting view timeout to 75000 [DEBUG 19.58 cbsdk.driver driver.py:48] NEWHANDLE:20 @20 (OK) {} [DEBUG 19.58 cbsdk.driver driver.py:39] MC_DS_GET:21 @1 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.59 cbsdk.driver driver.py:39] MC_DS_GET:22 @3 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.59 cbsdk.driver driver.py:39] MC_DS_GET:23 @5 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.59 cbsdk.driver driver.py:39] MC_DS_GET:24 @7 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.59 cbsdk.driver driver.py:39] MC_DS_GET:25 @9 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.59 cbsdk.driver driver.py:39] MC_DS_GET:26 @11 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.60 cbsdk.driver driver.py:39] MC_DS_GET:27 @13 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.60 cbsdk.driver driver.py:39] MC_DS_GET:28 @15 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.60 cbsdk.driver driver.py:39] MC_DS_GET:29 @17 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.60 cbsdk.driver driver.py:39] MC_DS_GET:30 @19 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.61 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:31 @2 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.61 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:32 @4 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.61 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:33 @6 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.62 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:34 @8 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.62 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:35 @10 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.62 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:36 @12 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.62 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:37 @14 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.63 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:38 @16 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.63 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:39 @18 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [DEBUG 19.63 cbsdk.driver driver.py:39] MC_DS_MUTATE_SET:40 @20 {'DSType': 'DSTYPE_SEEDED', 'DS': {'Count': 1000, 'Repeat': '_REP_', 'VSeed': 'SimpleValue_', 'KSize': 12, 'Continuous': True, 'VSize': 100, 'KSeed': 'SimpleKey_'}, 'Options': {'TimeRes': 1, 'IterWait': True, 'DelayMin': 0, 'DelayMax': 0}} [INFO 19.63 cbsdk.scenario base.py:51] Ramp-up for 5 seconds [PHASE 19.63 cbsdk.scenario logger.py:272] Changing phase to: RAMP [PHASE 24.64 cbsdk.scenario logger.py:275] Changing phase to: CHANGE [ACTION 24.64 cbsdk.scenario failover.py:178] Turning off service on ip:10.3.121.207 ssh_username:root [INFO 24.64 cbsdk.cluster.node node.py:135] --START: Commands: ['/etc/init.d/couchbase-server stop'] [SDKD(WARNING) 25.65 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 25.65 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to exception on {QA sa=/10.3.121.207:11210, #Rops=20, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 18734 Key: SimpleKey__REP_944, topWop=null, toWrite=0, interested=1} [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] java.io.IOException: Connection reset by peer [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.FileDispatcher.read0(Native Method) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.IOUtil.read(IOUtil.java:204) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:512) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:454) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=20, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 18734 Key: SimpleKey__REP_944, topWop=null, toWrite=0, interested=1}, attempt 0. [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18734 Key: SimpleKey__REP_944 [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18735 Key: SimpleKey__REP_944 [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18739 Key: SimpleKey__REP_945 [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] WARNING: Unknown exception encountered (for operation) future warnings will be suppressed [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] java.util.concurrent.CancellationException: Cancelled [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:170) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) [SDKD(WARNING) 25.66 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:114) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:131) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:64) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18740 Key: SimpleKey__REP_945 [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18741 Key: SimpleKey__REP_945 [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18766 Key: SimpleKey__REP_930 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18767 Key: SimpleKey__REP_930 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] WARNING: Unknown exception encountered (for operation) future warnings will be suppressed [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] java.util.concurrent.CancellationException: Cancelled [SDKD(WARNING) 25.67 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:170) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:114) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:131) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:68) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18770 Key: SimpleKey__REP_928 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18771 Key: SimpleKey__REP_941 [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18772 Key: SimpleKey__REP_941 [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18773 Key: SimpleKey__REP_941 [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.68 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18775 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18776 Key: SimpleKey__REP_936 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18777 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18778 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18783 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 0 Opaque: 18784 Key: SimpleKey__REP_941 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18785 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18786 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:40 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend [SDKD(WARNING) 25.69 cbsdk.sdkd.local executor.py:66] WARNING: Discarding partially completed op: Cmd: 1 Opaque: 18787 Key: SimpleKey__REP_936 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 [SDKD(WARNING) 27.66 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:42 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 27.66 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:42 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@7297a1fb [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:42 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to failure to connect to {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] java.net.ConnectException: Connection refused [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:399) [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:42 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 27.67 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 1. [SDKD(WARNING) 31.67 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:46 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 31.67 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:46 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@49c54f01 [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:46 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to failure to connect to {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] java.net.ConnectException: Connection refused [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:399) [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:46 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 31.68 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 2. [SDKD(WARNING) 31.95 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:46 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 31.95 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 3. [SDKD(WARNING) 31.96 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:46 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 31.96 cbsdk.sdkd.local executor.py:66] INFO: The channel or socket was null for {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [INFO 32.89 cbsdk.scenario failover.py:195] Not failing over (no_fo) [INFO 32.89 cbsdk.scenario failover.py:198] Sleeping for 60 seconds after failover [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:54 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:54 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@49e3c29e [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:54 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to failure to connect to {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] java.net.ConnectException: Connection refused [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:399) [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:28:54 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 39.68 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 4. [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:02 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:02 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@5d282c43 [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:02 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to failure to connect to {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] java.net.ConnectException: Connection refused [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [SDKD(WARNING) 47.96 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) [SDKD(WARNING) 47.97 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:399) [SDKD(WARNING) 47.97 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 47.97 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 47.97 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:02 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 47.97 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 5. [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:24 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:24 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@7614ca72 [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:24 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to failure to connect to {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] java.net.ConnectException: Connection refused [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:399) [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:24 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 69.69 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 6. [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:32 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:32 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@4ec9cff9 [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:32 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting due to failure to connect to {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] java.net.ConnectException: Connection refused [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:399) [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:247) [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:32 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 77.97 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 7. [ACTION 92.91 cbsdk.scenario failover.py:203] Turning on service on ip:10.3.121.207 ssh_username:root [INFO 92.91 cbsdk.cluster.node node.py:146] --START: ['/etc/init.d/couchbase-server start'] [INFO 94.20 cbsdk.cluster.ssh ssh.py:108] outbuf closed [PHASE 94.20 cbsdk.scenario logger.py:278] Changing phase to: REBOUND [ACTION 94.20 cbsdk.scenario base.py:59] Sleeping 90 seconds before collection [SDKD(WARNING) 98.06 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:52 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 98.06 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 8. [SDKD(WARNING) 98.06 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:52 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 98.06 cbsdk.sdkd.local executor.py:66] INFO: The channel or socket was null for {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 98.20 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:53 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 98.20 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 9. [SDKD(WARNING) 98.20 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:53 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 98.20 cbsdk.sdkd.local executor.py:66] INFO: The channel or socket was null for {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 99.60 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:54 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 99.60 cbsdk.sdkd.local executor.py:66] WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0}, attempt 10. [SDKD(WARNING) 99.60 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:54 AM net.spy.memcached.MemcachedConnection queueReconnect [SDKD(WARNING) 99.60 cbsdk.sdkd.local executor.py:66] INFO: The channel or socket was null for {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 99.69 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:54 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 99.69 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} [SDKD(WARNING) 99.70 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:29:54 AM net.spy.memcached.MemcachedConnection handleIO [SDKD(WARNING) 99.70 cbsdk.sdkd.local executor.py:66] INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@46f1324d [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:30:02 AM net.spy.memcached.MemcachedConnection attemptReconnects [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] INFO: Reconnecting {QA sa=/10.3.121.207:11210, #Rops=3, #Wops=0, #iq=0, topRop=Cmd: 0 Opaque: 407778 Key: SimpleKey__REP_20413, topWop=null, toWrite=0, interested=1} [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] Exception in thread "Memcached IO over {MemcachedConnection to /10.3.121.212:11210 /10.3.121.209:11210 /10.3.121.208:11210 /10.3.121.207:11210}" java.lang.AssertionError: Attempting to overwrite channel [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.protocol.TCPMemcachedNodeImpl.setChannel(TCPMemcachedNodeImpl.java:496) [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.protocol.TCPMemcachedNodeImpl.registerChannel(TCPMemcachedNodeImpl.java:484) [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.attemptReconnects(MemcachedConnection.java:680) [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:262) [SDKD(WARNING) 107.98 cbsdk.sdkd.local executor.py:66] at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:158) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-3: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-19: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.48 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-10: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-8: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.49 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-16: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.50 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-1: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-13: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-7: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.51 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-5: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-18: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.52 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-2: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-9: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.53 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-6: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-4: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-15: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.54 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-14: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-17: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.55 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-12: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-11: CMD [MC_DS_GET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:906) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncGets(MemcachedClient.java:920) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.GetCommandContext.doOneCommand(GetCommandContext.java:60) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] Exception in thread "SDK Handle-20: CMD [MC_DS_MUTATE_SET] " java.lang.AssertionError: IO Thread is not running. [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.checkState(MemcachedConnection.java:905) [SDKD(WARNING) 110.56 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedConnection.enqueueOperation(MemcachedConnection.java:719) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.asyncStore(MemcachedClient.java:302) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at net.spy.memcached.MemcachedClient.set(MemcachedClient.java:745) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:41) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:276) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:311) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:135) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:156) [SDKD(WARNING) 110.57 cbsdk.sdkd.local executor.py:66] at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:212) [PHASE 184.29 cbsdk.scenario logger.py:281] Changing phase to: DONE [DEBUG 184.29 cbsdk.driver driver.py:39] CANCEL:41 @1 {} [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.29 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.30 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.SdkServer shutdown [SDKD(WARNING) 184.31 cbsdk.sdkd.local executor.py:66] WARNING: Shutting Down.. [DEBUG 184.31 cbsdk.driver driver.py:48] CANCEL:41 @1 (OK) {} [DEBUG 184.31 cbsdk.driver driver.py:39] CANCEL:42 @3 {} [DEBUG 184.31 cbsdk.driver driver.py:48] CANCEL:42 @3 (OK) {} [DEBUG 184.31 cbsdk.driver driver.py:39] CANCEL:43 @5 {} [DEBUG 184.32 cbsdk.driver driver.py:48] CANCEL:43 @5 (OK) {} [DEBUG 184.32 cbsdk.driver driver.py:39] CANCEL:44 @7 {} [DEBUG 184.32 cbsdk.driver driver.py:48] CANCEL:44 @7 (OK) {} [DEBUG 184.32 cbsdk.driver driver.py:39] CANCEL:45 @9 {} [DEBUG 184.32 cbsdk.driver driver.py:48] CANCEL:45 @9 (OK) {} [DEBUG 184.32 cbsdk.driver driver.py:39] CANCEL:46 @11 {} [DEBUG 184.32 cbsdk.driver driver.py:48] CANCEL:46 @11 (OK) {} [DEBUG 184.32 cbsdk.driver driver.py:39] CANCEL:47 @13 {} [DEBUG 184.32 cbsdk.driver driver.py:48] CANCEL:47 @13 (OK) {} [DEBUG 184.32 cbsdk.driver driver.py:39] CANCEL:48 @15 {} [DEBUG 184.33 cbsdk.driver driver.py:48] CANCEL:48 @15 (OK) {} [DEBUG 184.33 cbsdk.driver driver.py:39] CANCEL:49 @17 {} [DEBUG 184.33 cbsdk.driver driver.py:48] CANCEL:49 @17 (OK) {} [DEBUG 184.33 cbsdk.driver driver.py:39] CANCEL:50 @19 {} [DEBUG 184.33 cbsdk.driver driver.py:48] CANCEL:50 @19 (OK) {} [DEBUG 184.33 cbsdk.driver driver.py:39] CANCEL:51 @2 {} [DEBUG 184.33 cbsdk.driver driver.py:48] CANCEL:51 @2 (OK) {} [DEBUG 184.33 cbsdk.driver driver.py:39] CANCEL:52 @4 {} [DEBUG 184.33 cbsdk.driver driver.py:48] CANCEL:52 @4 (OK) {} [DEBUG 184.33 cbsdk.driver driver.py:39] CANCEL:53 @6 {} [DEBUG 184.33 cbsdk.driver driver.py:48] CANCEL:53 @6 (OK) {} [DEBUG 184.33 cbsdk.driver driver.py:39] CANCEL:54 @8 {} [DEBUG 184.34 cbsdk.driver driver.py:48] CANCEL:54 @8 (OK) {} [DEBUG 184.34 cbsdk.driver driver.py:39] CANCEL:55 @10 {} [DEBUG 184.34 cbsdk.driver driver.py:48] CANCEL:55 @10 (OK) {} [DEBUG 184.34 cbsdk.driver driver.py:39] CANCEL:56 @12 {} [DEBUG 184.34 cbsdk.driver driver.py:48] CANCEL:56 @12 (OK) {} [DEBUG 184.34 cbsdk.driver driver.py:39] CANCEL:57 @14 {} [DEBUG 184.34 cbsdk.driver driver.py:48] CANCEL:57 @14 (OK) {} [DEBUG 184.34 cbsdk.driver driver.py:39] CANCEL:58 @16 {} [DEBUG 184.34 cbsdk.driver driver.py:48] CANCEL:58 @16 (OK) {} [DEBUG 184.34 cbsdk.driver driver.py:39] CANCEL:59 @18 {} [DEBUG 184.34 cbsdk.driver driver.py:48] CANCEL:59 @18 (OK) {} [DEBUG 184.35 cbsdk.driver driver.py:39] CANCEL:60 @20 {} [DEBUG 184.35 cbsdk.driver driver.py:48] CANCEL:60 @20 (OK) {} ./stester:235: UserWarning: Got exception running the test warn("Got exception running the test") [ERROR 184.35 cbsdk.__main__ stester:236] Couldn't decode (No JSON object could be decoded) Traceback (most recent call last): File "./stester", line 226, in run_test obj.run() File "/home/deeptida/testing/cbsdkd/lib/cbsdk/scenarios/failover.py", line 208, in run workload.collect() File "/home/deeptida/testing/cbsdkd/lib/cbsdk/workloads/getset.py", line 228, in collect resp = h.conduit.recv_msg() File "./lib/cbsdk/driver.py", line 47, in recv_msg msg = Res.Response.parse(txt) File "./lib/cbsdk/response.py", line 147, in parse txt, e)) ProtocolError: Couldn't decode (No JSON object could be decoded) [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:61 @1 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:62 @2 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:63 @3 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:64 @4 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:65 @5 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:66 @6 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:67 @7 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:68 @8 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:69 @9 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:70 @10 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:71 @11 {} [DEBUG 184.35 cbsdk.driver driver.py:39] CLOSEHANDLE:72 @12 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:73 @13 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:74 @14 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:75 @15 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:76 @16 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:77 @17 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:78 @18 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:79 @19 {} [DEBUG 184.36 cbsdk.driver driver.py:39] CLOSEHANDLE:80 @20 {} [DEBUG 184.36 cbsdk.driver driver.py:39] GOODBYE:0 @Control {} --- Done --- [SDKD(WARNING) 184.36 cbsdk.sdkd.local executor.py:66] Feb 21, 2013 2:31:19 AM com.couchbase.sdkd.server.ControlServer run Output now empty. Breaking