deeptida@ubu-1512:~/new_tests/cbsdkd$ ./logview.py -f deepti.log Opening deepti.log Log version 1.2.0 detected Verbatim Commandline -i /home/deeptida/new_tests/mark-vm.ini \ -c failover.Once \ --dsw_timeres 1 \ --cl_bucket_type memcached \ --cl_bucket_password verysecret \ --cl_bucket_name a_memcached_bucket \ -I /home/deeptida/new_tests/sdkd-cluster.args \ --portservice 8091 \ --portaction FW_DROP \ -o deepti.log \ Effective stester command line -C share/rexec \ --rexec_path /home/deeptida/new_tests/sdkd-java/run-sdkd-java \ --rexec_port 8050 \ -i /home/deeptida/new_tests/mark-vm.ini \ -c failover.Once \ --dsw_timeres 1 \ --cl_bucket_type memcached \ --cl_bucket_password verysecret \ --cl_bucket_name a_memcached_bucket \ --portservice 8091 \ --portaction FW_DROP \ -o deepti.log \ --- Effective Test Configuration { 'action': 'FO_NOACTION', 'action_delay': 0, 'cl_bucket_add_dfl': True, 'cl_bucket_name': 'a_memcached_bucket', 'cl_bucket_password': 'verysecret', 'cl_bucket_ram': 512, 'cl_bucket_replicas': 1, 'cl_bucket_type': 'memcached', 'cl_fake': False, 'cl_nodes_ok': False, 'cl_noinit': False, 'cl_replicas': 1, 'cl_use_backups': False, 'dsw_continuous': True, 'dsw_delay': -1, 'dsw_delay_max': 10, 'dsw_delay_min': 1, 'dsw_iterwait': 1, 'dsw_kseed': 'SimpleKey_', 'dsw_ksize': 12, 'dsw_kvcount': 15000, 'dsw_nthreads': 10, 'dsw_opmode': 'both', 'dsw_persist': 0, 'dsw_replicate': 0, 'dsw_timeres': 1, 'dsw_ttl': 0, 'dsw_vsize': 100, 'ept': False, 'grace_after': 10, 'no_fo': False, 'node': [ u'10.3.121.207', u'10.3.121.209', u'10.3.121.208', u'10.3.121.212'], 'nodes_rest_password': '123456', 'nodes_rest_port': 8091, 'nodes_rest_user': 'Administrator', 'nodes_ssh_disable': False, 'nodes_ssh_key': None, 'nodes_ssh_password': 'root', 'nodes_ssh_user': 'root', 'num_nodes': 1, 'portaction': 'FW_DROP', 'portservice': 8091, 'ramp': 5, 'rebound': 90, 'rexec_argfile': None, 'rexec_args': None, 'rexec_password': None, 'rexec_path': '/home/deeptida/new_tests/sdkd-java/run-sdkd-java', 'rexec_port': 8050, 'rexec_wait': 2, 'service': None, 'svc_itergrace': 0, 'svc_itergrace_off': 0, 'svc_itergrace_on': 0, 'svcaction': 'RESTART'} --- Found server version 2.0.1-173-rel-enterprise --- Found SDK Information: { 'CAPS': { 'CANCEL': True, 'CONTINUOUS': True, 'DS_SHARED': True, 'OBS': True, 'PREAMBLE': False, 'TTL': True, 'VIEWS': True}, 'COMPONENTS': { 'SDK': 'Couchbase Java Client 1.1.5\n\nTree Version: 1.1.5-6-g7ee8289\nLast Commit ID: 7ee82891a4f3c1e2759233bff9b2d28e11ccf0b2\n\nCompiled by deeptida@ubu-1512 on Tue May 14 00:59:36 PDT 2013', 'SPY': 'Spymemcached 2.8.12\n\nTree Version: 2.8.12-9-g036081f\nLast Commit ID: 036081f046261c2f75eee6414bfacc915f450dda\n\nCompiled by deeptida@ubu-1512 on Tue May 14 01:00:00 PDT 2013'}} Will show phase timings.. -------------------------- Phase statistics for RAMP OK/sec: 1538 OK: 7693 ERR: 0 Phase statistics for CHANGE OK/sec: 2503 OK: 27534 ERR: 7 Phase statistics for REBOUND OK/sec: 3258 OK: 293237 ERR: 0 --------------------- Detailed Timeline.. +0s SDKD: May 14, 2013 1:01:38 AM com.couchbase.client.CouchbaseProperties setPropertyFile SDKD: INFO: Could not load properties file "cbclient.properties" because: File not found with system classloader. SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection createConnections SDKD: INFO: Added {QA sa=/10.3.121.212:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection createConnections SDKD: INFO: Added {QA sa=/10.3.121.209:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection createConnections SDKD: INFO: Added {QA sa=/10.3.121.208:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection createConnections SDKD: INFO: Added {QA sa=/10.3.121.207:11210, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection handleIO SDKD: INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@351563ff SDKD: May 14, 2013 1:01:38 AM com.couchbase.client.CouchbaseClient SDKD: INFO: viewmode property isn't defined. Setting viewmode to production mode SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection handleIO SDKD: INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@a210b5b SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection handleIO SDKD: INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@7f70888e SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.MemcachedConnection handleIO SDKD: INFO: Connection state changed for sun.nio.ch.SelectionKeyImpl@f581593 SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.auth.AuthThread$1 receivedStatus SDKD: INFO: Authenticated to /10.3.121.212:11210 SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.auth.AuthThread$1 receivedStatus SDKD: INFO: Authenticated to /10.3.121.209:11210 SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.auth.AuthThread$1 receivedStatus SDKD: INFO: Authenticated to /10.3.121.208:11210 SDKD: May 14, 2013 1:01:38 AM net.spy.memcached.auth.AuthThread$1 receivedStatus SDKD: INFO: Authenticated to /10.3.121.207:11210 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Created new connection to 8050 DEFAULT: Ramp-up for 5 seconds ACTION: Changing phase to: RAMP OK: 2341 ERR: 0 +2s OK: 3325 ERR: 0 +4s ACTION: Changing phase to: CHANGE ACTION: Turning off service on ip:10.3.121.207 ssh_username:root ACTION: Failing over nodes [ip:10.3.121.207 ssh_username:root] ACTION: Failing over node ip:10.3.121.207 ssh_username:root OK: 4394 ERR: 0 +6s DEFAULT: fail_over successful ACTION: No post-failover action requested DEFAULT: Sleeping for 10 seconds after failover SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.MemcachedConnection handleIO SDKD: INFO: Reconnecting due to exception on {QA sa=/10.3.121.207:11210, #Rops=6, #Wops=1, #iq=3, topRop=Cmd: 1 Opaque: 10581 Key: SimpleKey__REP_512 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, topWop=Cmd: 1 Opaque: 10599 Key: SimpleKey__REP_534 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, toWrite=0, interested=5} SDKD: java.io.IOException: Connection reset by peer SDKD: at sun.nio.ch.FileDispatcher.read0(Native Method) SDKD: at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) SDKD: at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237) SDKD: at sun.nio.ch.IOUtil.read(IOUtil.java:204) SDKD: at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) SDKD: at net.spy.memcached.MemcachedConnection.handleReads(MemcachedConnection.java:515) SDKD: at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:457) SDKD: at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:250) SDKD: at com.couchbase.client.CouchbaseMemcachedConnection.run(CouchbaseMemcachedConnection.java:153) SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.MemcachedConnection queueReconnect SDKD: WARNING: Closing, and reopening {QA sa=/10.3.121.207:11210, #Rops=6, #Wops=1, #iq=3, topRop=Cmd: 1 Opaque: 10581 Key: SimpleKey__REP_512 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, topWop=Cmd: 1 Opaque: 10599 Key: SimpleKey__REP_534 Cas: 0 Exp: 0 Flags: 0 Data Length: 100, toWrite=0, interested=5}, attempt 0. SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 1 Opaque: 10581 Key: SimpleKey__REP_512 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 SDKD: May 14, 2013 1:01:44 AM com.couchbase.sdkd.cbclient.CommandResult warnAbout SDKD: WARNING: Unknown exception encountered (for operation) future warnings will be suppressed SDKD: java.util.concurrent.CancellationException: Cancelled SDKD: at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:170) SDKD: at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:133) SDKD: at com.couchbase.sdkd.cbclient.PendingCommand.onReady(PendingCommand.java:55) SDKD: at com.couchbase.sdkd.cbclient.OperationCollector.submitSync(OperationCollector.java:118) SDKD: at com.couchbase.sdkd.cbclient.OperationCollector.submit(OperationCollector.java:135) SDKD: at com.couchbase.sdkd.cbclient.CommandContext.execIter(CommandContext.java:318) SDKD: at com.couchbase.sdkd.cbclient.CommandContext.execute(CommandContext.java:374) SDKD: at com.couchbase.sdkd.server.SdkServer.executeCommand(SdkServer.java:168) SDKD: at com.couchbase.sdkd.server.SdkServer.handleRequest(SdkServer.java:189) SDKD: at com.couchbase.sdkd.server.SdkServer.run(SdkServer.java:245) SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 1 Opaque: 10586 Key: SimpleKey__REP_534 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 0 Opaque: 10588 Key: SimpleKey__REP_534 SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 1 Opaque: 10591 Key: SimpleKey__REP_513 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 0 Opaque: 10593 Key: SimpleKey__REP_531 SDKD: May 14, 2013 1:01:44 AM net.spy.memcached.protocol.TCPMemcachedNodeImpl setupResend SDKD: WARNING: Discarding partially completed op: Cmd: 1 Opaque: 10594 Key: SimpleKey__REP_531 Cas: 0 Exp: 0 Flags: 0 Data Length: 100 SDKD: May 14, 2013 1:01:44 AM com.couchbase.client.CouchbaseMemcachedConnection reconfigure SDKD: INFO: Scheduling Node /10.3.121.207:11210for shutdown. OK: 5209 ERR: 6 +8s OK: 4483 ERR: 1 +10s OK: 7364 ERR: 0 +12s OK: 3243 ERR: 0 +14s OK: 4868 ERR: 0 +16s ACTION: Turning on service on ip:10.3.121.207 ssh_username:root DEFAULT: Skipping '-A INPUT -p tcp -m comment --comment "CB_BLOCK" -m tcp --dport 8091 -j DROP ' DEFAULT: Lines passed to ipt restore: # Generated by iptables-save v1.3.5 on Tue May 14 01:00:17 2013 *nat :PREROUTING ACCEPT [67:8905] :POSTROUTING ACCEPT [37:2230] :OUTPUT ACCEPT [37:2230] COMMIT # Completed on Tue May 14 01:00:17 2013 # Generated by iptables-save v1.3.5 on Tue May 14 01:00:17 2013 *filter :INPUT ACCEPT [9764:2657170] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [10753:2602695] :RH-Firewall-1-INPUT - [0:0] COMMIT # Completed on Tue May 14 01:00:17 2013 DEFAULT: Collected EC=0 ACTION: Changing phase to: REBOUND ACTION: Sleeping 90 seconds before collection OK: 7380 ERR: 0 +18s OK: 6281 ERR: 0 +20s OK: 7657 ERR: 0 +22s OK: 4630 ERR: 0 +24s OK: 5746 ERR: 0 +26s OK: 6826 ERR: 0 +28s OK: 5727 ERR: 0 +30s OK: 7195 ERR: 0 +32s OK: 4641 ERR: 0 +34s OK: 6797 ERR: 0 +36s OK: 6370 ERR: 0 +38s OK: 6262 ERR: 0 +40s OK: 7064 ERR: 0 +42s OK: 4135 ERR: 0 +44s OK: 7557 ERR: 0 +46s OK: 7179 ERR: 0 +48s OK: 7014 ERR: 0 +50s OK: 7700 ERR: 0 +52s OK: 3225 ERR: 0 +54s OK: 5291 ERR: 0 +56s OK: 7198 ERR: 0 +58s OK: 6583 ERR: 0 +60s OK: 7168 ERR: 0 +62s OK: 5539 ERR: 0 +64s OK: 7039 ERR: 0 +66s OK: 6238 ERR: 0 +68s OK: 6353 ERR: 0 +70s OK: 7131 ERR: 0 +72s OK: 4269 ERR: 0 +74s OK: 7211 ERR: 0 +76s OK: 6754 ERR: 0 +78s OK: 6240 ERR: 0 +80s OK: 6914 ERR: 0 +82s OK: 3853 ERR: 0 +84s OK: 6007 ERR: 0 +86s OK: 7582 ERR: 0 +88s OK: 7055 ERR: 0 +90s OK: 7367 ERR: 0 +92s OK: 5622 ERR: 0 +94s OK: 7670 ERR: 0 +96s OK: 8857 ERR: 0 +98s OK: 7616 ERR: 0 +100s OK: 8418 ERR: 0 +102s OK: 6242 ERR: 0 +104s OK: 7634 ERR: 0 +106s ACTION: Changing phase to: DONE SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown SDKD: WARNING: Shutting Down.. SDKD: May 14, 2013 1:03:25 AM com.couchbase.sdkd.server.SdkServer shutdown OK: 1050 ERR: 0 MC_DS_GET:21 @1 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17572, '520': 21}} MC_DS_GET:22 @3 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17051, '520': 543}} MC_DS_GET:23 @5 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 9166, '520': 8480}} MC_DS_GET:24 @7 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17445, '520': 151}} MC_DS_GET:25 @9 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 13147, '520': 4551}} MC_DS_GET:26 @11 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17518, '520': 28}} MC_DS_GET:27 @13 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 13210, '520': 4509}} MC_DS_GET:28 @15 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 16888, '520': 725}} MC_DS_GET:29 @17 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 15558, '520': 2118, '257': 1}} MC_DS_GET:30 @19 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 16818, '520': 766, '257': 1}} MC_DS_MUTATE_SET:31 @2 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17635}} MC_DS_MUTATE_SET:32 @4 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17418, '257': 1}} MC_DS_MUTATE_SET:33 @6 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17567, '257': 1}} MC_DS_MUTATE_SET:34 @8 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17536}} MC_DS_MUTATE_SET:35 @10 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17497, '257': 1}} MC_DS_MUTATE_SET:36 @12 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17295, '516': 1}} MC_DS_MUTATE_SET:37 @14 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17456}} MC_DS_MUTATE_SET:38 @16 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17602}} MC_DS_MUTATE_SET:39 @18 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17570, '257': 1}} MC_DS_MUTATE_SET:40 @20 (OK) {'Timings': '< Timings Truncated >', 'Summary': {'0': 17565}}