Opening deepti.log Log version 1.2.0 detected Verbatim Commandline -i /home/deeptida/new_tests/mark-vm.ini \ -c failover.Once \ --dsw_timeres 1 \ --cl_bucket_type memcached \ --cl_bucket_password verysecret \ --cl_bucket_name a_memcached_bucket \ -I /home/deeptida/new_tests/sdkd-cluster.args \ --portservice 8091 \ --portaction FW_DROP \ -o deepti.log \ Effective stester command line -C share/rexec \ --rexec_path /home/deeptida/new_tests/sdkd-java/run-sdkd-java \ --rexec_port 8050 \ -i /home/deeptida/new_tests/mark-vm.ini \ -c failover.Once \ --dsw_timeres 1 \ --cl_bucket_type memcached \ --cl_bucket_password verysecret \ --cl_bucket_name a_memcached_bucket \ --portservice 8091 \ --portaction FW_DROP \ -o deepti.log \ --- Effective Test Configuration { 'action': 'FO_NOACTION', 'action_delay': 0, 'cl_bucket_add_dfl': True, 'cl_bucket_name': 'a_memcached_bucket', 'cl_bucket_password': 'verysecret', 'cl_bucket_ram': 512, 'cl_bucket_replicas': 1, 'cl_bucket_type': 'memcached', 'cl_fake': False, 'cl_nodes_ok': False, 'cl_noinit': False, 'cl_replicas': 1, 'cl_use_backups': False, 'dsw_continuous': True, 'dsw_delay': -1, 'dsw_delay_max': 10, 'dsw_delay_min': 1, 'dsw_iterwait': 1, 'dsw_kseed': 'SimpleKey_', 'dsw_ksize': 12, 'dsw_kvcount': 15000, 'dsw_nthreads': 10, 'dsw_opmode': 'both', 'dsw_persist': 0, 'dsw_replicate': 0, 'dsw_timeres': 1, 'dsw_ttl': 0, 'dsw_vsize': 100, 'ept': False, 'grace_after': 10, 'no_fo': False, 'node': [ u'10.3.121.207', u'10.3.121.209', u'10.3.121.208', u'10.3.121.212'], 'nodes_rest_password': '123456', 'nodes_rest_port': 8091, 'nodes_rest_user': 'Administrator', 'nodes_ssh_disable': False, 'nodes_ssh_key': None, 'nodes_ssh_password': 'root', 'nodes_ssh_user': 'root', 'num_nodes': 1, 'portaction': 'FW_DROP', 'portservice': 8091, 'ramp': 5, 'rebound': 90, 'rexec_argfile': None, 'rexec_args': None, 'rexec_password': None, 'rexec_path': '/home/deeptida/new_tests/sdkd-java/run-sdkd-java', 'rexec_port': 8050, 'rexec_wait': 2, 'service': None, 'svc_itergrace': 0, 'svc_itergrace_off': 0, 'svc_itergrace_on': 0, 'svcaction': 'RESTART'} --- Found server version 2.0.1-173-rel-enterprise --- Found SDK Information: { 'CAPS': { 'CANCEL': True, 'CONTINUOUS': True, 'DS_SHARED': True, 'OBS': True, 'PREAMBLE': False, 'TTL': True, 'VIEWS': True}, 'COMPONENTS': { 'SDK': 'Couchbase Java Client 1.1.5\n\nTree Version: 1.1.5\nLast Commit ID: 8bf710d0f1560c982e545458e83c7a6f27921d89\n\nCompiled by deeptida@ubu-1512 on Fri May 10 03:30:30 PDT 2013', 'SPY': 'Spymemcached 2.8.12\n\nTree Version: 2.8.12-9-g036081f\nLast Commit ID: 036081f046261c2f75eee6414bfacc915f450dda\n\nCompiled by deeptida@ubu-1512 on Fri May 10 03:30:45 PDT 2013'}} Will show phase timings.. -------------------------- Phase statistics for RAMP OK/sec: 932 OK: 4660 ERR: 6279 Phase statistics for CHANGE OK/sec: 1581 OK: 15818 ERR: 19639 Phase statistics for REBOUND OK/sec: 1352 OK: 121697 ERR: 193026 --------------------- Detailed Timeline.. +0s DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 SDKD: May 14, 2013 12:55:07 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout SDKD: WARNING: Unknown exception encountered (for operation) future warnings will be suppressed SDKD: java.lang.ClassCastException: com.couchbase.client.CouchbaseMemcachedConnection cannot be cast to com.couchbase.client.CouchbaseConnection SDKD: at com.couchbase.client.CouchbaseClient.observePoll(CouchbaseClient.java:1804) SDKD: at com.couchbase.client.CouchbaseClient.set(CouchbaseClient.java:1233) SDKD: at com.couchbase.sdkd.cbclient.durability.ObsEnabled.mutate(ObsEnabled.java:41) SDKD: at com.couchbase.sdkd.cbclient.SetCommandContext.doOneCommand(SetCommandContext.java:37) SDKD: at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:303) SDKD: at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:374) SDKD: at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:168) SDKD: at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:189) SDKD: at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:245) DEFAULT: Ramp-up for 5 seconds ACTION: Changing phase to: RAMP OK: 1648 ERR: 2149 +2s OK: 1831 ERR: 2669 +4s ACTION: Changing phase to: CHANGE ACTION: Turning off service on ip:10.3.121.207 ssh_username:root ACTION: Failing over nodes [ip:10.3.121.207 ssh_username:root] ACTION: Failing over node ip:10.3.121.207 ssh_username:root DEFAULT: fail_over successful ACTION: No post-failover action requested DEFAULT: Sleeping for 10 seconds after failover SDKD: May 14, 2013 12:55:12 AM net.spy.memcached.MemcachedConnection handleIO SDKD: INFO: Reconnecting due to exception on {QA sa=/10.3.121.207:11210, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 13687 Key: SimpleKey__REP_651 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, topWop=null, toWrite=0, interested=1} SDKD: java.io.IOException: Disconnected unexpected, will reconnect. SDKD: at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:529) SDKD: at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:457) SDKD: at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250) SDKD: at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:153) SDKD: May 14, 2013 12:55:12 AM net.spy.memcached.MemcachedConnection queueReconnect SDKD: WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=1, #Wops=0, #iq=0, topRop=Cmd: 1 Opaque: 13687 Key: SimpleKey__REP_651 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, topWop=null, toWrite=0, interested=1}, attempt 0. SDKD: May 14, 2013 12:55:12 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 1 Opaque: 13687 Key: SimpleKey__REP_651 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 SDKD: May 14, 2013 12:55:12 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout SDKD: WARNING: Unknown exception encountered (for operation) future warnings will be suppressed SDKD: java.util.concurrent.CancellationException: Cancelled SDKD: at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:170) SDKD: at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) SDKD: at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55) SDKD: at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:118) SDKD: at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:135) SDKD: at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:318) SDKD: at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:374) SDKD: at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:168) SDKD: at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:189) SDKD: at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:245) SDKD: May 14, 2013 12:55:12 AM com.couchbase.client.CouchbaseMemcachedConnection reconfigure SDKD: INFO: Scheduling Node /10.3.121.207:11210for shutdown. OK: 2024 ERR: 3001 +6s OK: 1958 ERR: 3548 +8s SDKD: May 14, 2013 12:55:15 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout SDKD: WARNING: Unknown exception encountered (for operation) future warnings will be suppressed SDKD: java.lang.ClassCastException OK: 2647 ERR: 3860 +10s OK: 4329 ERR: 4406 +12s OK: 4138 ERR: 4188 +14s ACTION: Turning on service on ip:10.3.121.207 ssh_username:root DEFAULT: Skipping '-A INPUT -p tcp -m comment --comment "CB_BLOCK" -m tcp --dport 8091 -j DROP ' DEFAULT: Lines passed to ipt restore: # Generated by iptables-save v1.3.5 on Tue May 14 00:53:45 2013 *nat :PREROUTING ACCEPT [2555:659670] :POSTROUTING ACCEPT [1273:82121] :OUTPUT ACCEPT [1273:82121] COMMIT # Completed on Tue May 14 00:53:45 2013 # Generated by iptables-save v1.3.5 on Tue May 14 00:53:45 2013 *filter :INPUT ACCEPT [1739374:1083370093] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1727024:1058917975] :RH-Firewall-1-INPUT - [0:0] COMMIT # Completed on Tue May 14 00:53:45 2013 DEFAULT: Collected EC=0 ACTION: Changing phase to: REBOUND ACTION: Sleeping 90 seconds before collection OK: 3915 ERR: 4436 +16s OK: 3226 ERR: 4334 +18s OK: 2967 ERR: 4246 +20s OK: 3659 ERR: 4536 +22s OK: 3286 ERR: 4156 +24s OK: 3270 ERR: 4499 +26s OK: 2062 ERR: 4299 +28s OK: 1893 ERR: 4077 +30s OK: 2324 ERR: 4529 +32s OK: 3051 ERR: 4258 +34s OK: 3209 ERR: 4338 +36s OK: 3020 ERR: 4467 +38s OK: 2327 ERR: 4186 +40s OK: 2354 ERR: 4379 +42s OK: 1483 ERR: 4152 +44s OK: 979 ERR: 4435 +46s OK: 1100 ERR: 4330 +48s OK: 1287 ERR: 4119 +50s OK: 570 ERR: 4439 +52s OK: 424 ERR: 4163 +54s OK: 466 ERR: 4337 +56s OK: 666 ERR: 4440 +58s OK: 766 ERR: 3845 +60s OK: 821 ERR: 4354 +62s OK: 981 ERR: 4090 +64s OK: 1222 ERR: 4383 +66s OK: 993 ERR: 4362 +68s OK: 808 ERR: 4196 +70s OK: 1253 ERR: 4432 +72s OK: 3254 ERR: 4109 +74s OK: 4077 ERR: 4415 +76s OK: 4461 ERR: 4458 +78s OK: 4087 ERR: 4132 +80s OK: 4261 ERR: 4251 +82s OK: 3981 ERR: 3912 +84s OK: 4389 ERR: 4419 +86s OK: 4295 ERR: 4146 +88s OK: 3914 ERR: 3912 +90s OK: 4230 ERR: 4372 +92s OK: 4157 ERR: 4166 +94s OK: 4454 ERR: 4327 +96s OK: 4547 ERR: 4544 +98s OK: 4211 ERR: 4178 +100s OK: 4354 ERR: 4412 +102s OK: 4286 ERR: 4251 +104s ACTION: Changing phase to: DONE OK: 3457 ERR: 3545 +106s SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 12:56:53 AM com.couchbase.sdkd.server.SdkServer shutdown OK: 0 ERR: 0 MC_DS_GET:21 @1 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 11763, '520': 10329}} MC_DS_GET:22 @3 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 21628, '520': 349}} MC_DS_GET:23 @5 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 16527, '520': 5724}} MC_DS_GET:24 @7 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 9462, '520': 12655}} MC_DS_GET:25 @9 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 15242, '520': 7044}} MC_DS_GET:26 @11 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 11745, '520': 10529}} MC_DS_GET:27 @13 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 12429, '520': 9693}} MC_DS_GET:28 @15 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 15494, '520': 6789}} MC_DS_GET:29 @17 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 10098, '520': 12166}} MC_DS_GET:30 @19 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 18984, '520': 3072}} MC_DS_MUTATE_SET:31 @2 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 22022}} MC_DS_MUTATE_SET:32 @4 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 21950}} MC_DS_MUTATE_SET:33 @6 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 21931}} MC_DS_MUTATE_SET:34 @8 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 22103}} MC_DS_MUTATE_SET:35 @10 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 22049}} MC_DS_MUTATE_SET:36 @12 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 22106}} MC_DS_MUTATE_SET:37 @14 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 22176}} MC_DS_MUTATE_SET:38 @16 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 21977}} MC_DS_MUTATE_SET:39 @18 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 21924}} MC_DS_MUTATE_SET:40 @20 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'257': 21949}}