[2016-08-18 17:22:08,728] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:08,968] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.96
[2016-08-18 17:22:09,638] - [remote_util] [140513690150720] - INFO - 172.23.105.96 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:22:09,801] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:10,046] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.190
[2016-08-18 17:22:10,651] - [remote_util] [140513690150720] - INFO - 172.23.105.190 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:22:10,811] - [remote_util] [140513690150720] - INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:11,057] - [remote_util] [140513690150720] - INFO - Connected to 172.23.106.66
[2016-08-18 17:22:11,744] - [remote_util] [140513690150720] - INFO - 172.23.106.66 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:22:11,901] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:12,143] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.69
[2016-08-18 17:22:12,739] - [remote_util] [140513690150720] - INFO - 172.23.105.69 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:22:12,897] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:13,115] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.54
[2016-08-18 17:22:13,732] - [remote_util] [140513690150720] - INFO - 172.23.105.54 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:22:13,908] - [fts_base] [140513456174848] - INFO - ==== FTSbasetests setup is started for test #15 graceful_failover_and_delta_recovery_during_index_building ====
[2016-08-18 17:22:13,925] - [fts_base] [140513456174848] - INFO - removing nodes from cluster ...
[2016-08-18 17:22:13,934] - [fts_base] [140513456174848] - INFO - cleanup [ip:172.23.105.96 port:8091 ssh_username:root]
[2016-08-18 17:22:13,943] - [bucket_helper] [140513456174848] - INFO - deleting existing buckets [] on 172.23.105.96
[2016-08-18 17:22:13,968] - [cluster_helper] [140513456174848] - INFO - waiting for ns_server @ 172.23.105.96:8091
[2016-08-18 17:22:13,974] - [cluster_helper] [140513456174848] - INFO - ns_server @ 172.23.105.96:8091 is running
[2016-08-18 17:22:13,974] - [fts_base] [140513456174848] - INFO - Initializing Cluster ...
[2016-08-18 17:22:14,924] - [rest_client] [140513447782144] - INFO - pools/default params : memoryQuota=2069
[2016-08-18 17:22:14,931] - [rest_client] [140513447782144] - INFO - settings/indexes params : storageMode=forestdb
[2016-08-18 17:22:14,938] - [rest_client] [140513447782144] - INFO - settings/web params on 172.23.105.96:8091:username=Administrator&password=password&port=8091
[2016-08-18 17:22:14,962] - [fts_base] [140513456174848] - INFO - 172.23.105.190 will be configured with services kv,fts
[2016-08-18 17:22:15,963] - [task] [140513447782144] - INFO - adding node 172.23.105.190:8091 to cluster
[2016-08-18 17:22:15,963] - [rest_client] [140513447782144] - INFO - adding remote node @172.23.105.190:8091 to this cluster @172.23.105.96:8091
[2016-08-18 17:22:20,328] - [rest_client] [140513447782144] - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.96%2Cns_1%40172.23.105.190
[2016-08-18 17:22:20,340] - [rest_client] [140513447782144] - INFO - rebalance operation started
[2016-08-18 17:22:20,345] - [rest_client] [140513447782144] - INFO - rebalance percentage : 0.00 %
[2016-08-18 17:22:30,382] - [task] [140513447782144] - INFO - rebalancing was completed with progress: 100% in 10.0419890881 sec
[2016-08-18 17:22:31,402] - [rest_client] [140513447782144] - INFO - http://172.23.105.96:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1569&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None
[2016-08-18 17:22:31,415] - [rest_client] [140513447782144] - INFO - 0.01 seconds to create bucket default
[2016-08-18 17:22:31,415] - [bucket_helper] [140513447782144] - INFO - waiting for memcached bucket : default in 172.23.105.96 to accept set ops
[2016-08-18 17:22:33,122] - [data_helper] [140513447782144] - INFO - creating direct client 172.23.105.190:11210 default
[2016-08-18 17:22:33,237] - [data_helper] [140513447782144] - INFO - creating direct client 172.23.105.96:11210 default
[2016-08-18 17:22:33,418] - [data_helper] [140513447782144] - INFO - creating direct client 172.23.105.190:11210 default
[2016-08-18 17:22:33,543] - [data_helper] [140513447782144] - INFO - creating direct client 172.23.105.96:11210 default
[2016-08-18 17:22:34,342] - [task] [140513447782144] - INFO - bucket 'default' was created with per node RAM quota: 1569
[2016-08-18 17:22:34,386] - [rest_client] [140513456174848] - INFO - /diag/eval status on 172.23.105.96:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-08-18 17:22:34,388] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:34,668] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.96
[2016-08-18 17:22:35,246] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.96: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:35,353] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:35,354] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:35,354] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.96: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:35,462] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:35,462] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:35,579] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:35,853] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.190
[2016-08-18 17:22:36,414] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.190: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:37,201] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:37,202] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:37,203] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.190: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:37,917] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:37,918] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:38,035] - [remote_util] [140513456174848] - INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:38,278] - [remote_util] [140513456174848] - INFO - Connected to 172.23.106.66
[2016-08-18 17:22:38,811] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.106.66: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:38,933] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:38,935] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:38,935] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.106.66: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:39,050] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:39,051] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:39,167] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:39,444] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.69
[2016-08-18 17:22:40,025] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.69: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:40,143] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:40,144] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:40,145] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.69: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:40,259] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:40,259] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:40,263] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-08-18 17:22:40,478] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.54
[2016-08-18 17:22:41,052] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.54: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:41,205] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:41,206] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:41,206] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.54: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:22:41,348] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:22:41,350] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:22:41,464] - [fts_base] [140513456174848] - INFO - {'172.23.105.190': {'crash in forestdb': 0, 'panic:': 0}, '172.23.106.66': {'crash in forestdb': 0, 'panic:': 0}, '172.23.105.69': {'crash in forestdb': 0, 'panic:': 0}, '172.23.105.54': {'crash in forestdb': 0, 'panic:': 0}, '172.23.105.96': {'crash in forestdb': 0, 'panic:': 0}}
[2016-08-18 17:22:41,465] - [fts_base] [140513456174848] - INFO - ==== FTSbasetests setup is finished for test #15 graceful_failover_and_delta_recovery_during_index_building ====
[2016-08-18 17:22:46,428] - [data_helper] [140513456174848] - INFO - creating direct client 172.23.105.190:11210 default
[2016-08-18 17:22:46,735] - [data_helper] [140513456174848] - INFO - creating direct client 172.23.105.96:11210 default
[2016-08-18 17:22:59,119] - [fts_base] [140513456174848] - INFO - Loading phase complete!
[2016-08-18 17:22:59,149] - [fts_base] [140513456174848] - INFO - Checking if index already exists ...
[2016-08-18 17:22:59,174] - [rest_client] [140513456174848] - ERROR - http://172.23.105.190:8094/api/index/default_index_1 error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found 
 rest_auth: preparePerm, err: index not found 
[2016-08-18 17:22:59,178] - [rest_client] [140513456174848] - ERROR - http://172.23.105.190:8094/api/index/default_index_1 error 403 reason: status: 403, content: rest_auth: preparePerm, err: index not found 
 rest_auth: preparePerm, err: index not found 
[2016-08-18 17:22:59,178] - [fts_base] [140513456174848] - INFO - Creating fulltext-index default_index_1 on 172.23.105.190
[2016-08-18 17:22:59,178] - [rest_client] [140513456174848] - INFO - {"params": {}, "name": "default_index_1", "planParams": {"hierarchyRules": null, "nodePlanParams": null, "numReplicas": 0, "planFrozen": false, "maxPartitionsPerPIndex": 32}, "sourceName": "default", "sourceUUID": "", "sourceType": "couchbase", "type": "fulltext-index", "sourceParams": {"authUser": "default", "dataManagerSleepMaxMS": 20000, "authSaslUser": "", "clusterManagerSleepMaxMS": 20000, "authSaslPassword": "", "clusterManagerSleepInitMS": 0, "dataManagerBackoffFactor": 0, "authPassword": "", "dataManagerSleepInitMS": 0, "feedBufferAckThreshold": 0, "feedBufferSizeBytes": 0, "clusterManagerBackoffFactor": 0}, "uuid": ""}
[2016-08-18 17:22:59,237] - [rest_client] [140513456174848] - INFO - Index default_index_1 created
[2016-08-18 17:22:59,237] - [fts_base] [140513456174848] - INFO - Validating index distribution for default_index_1 ...
[2016-08-18 17:22:59,262] - [fts_base] [140513456174848] - INFO - sleep for 5 secs. No pindexes found, waiting for index to get created ...
[2016-08-18 17:23:04,401] - [fts_base] [140513456174848] - INFO - Validated: Number of PIndexes = 32
[2016-08-18 17:23:04,415] - [fts_base] [140513456174848] - INFO - Validated: Every pIndex serves 32 partitions or lesser
[2016-08-18 17:23:04,415] - [fts_base] [140513456174848] - INFO - Validated: pIndexes are distributed across [u'e6b2a25cc34d8ea4a349ad0981b179d6'] 
[2016-08-18 17:23:04,415] - [fts_base] [140513456174848] - INFO - Expecting num of partitions in each node in range 992-1024
[2016-08-18 17:23:04,415] - [fts_base] [140513456174848] - INFO - Validated: Node e6b2a25cc34d8ea4a349ad0981b179d6 houses 32 pindexes which serve 1024 partitions
[2016-08-18 17:23:04,416] - [fts_base] [140513456174848] - INFO - sleep for 10 secs.  ...
[2016-08-18 17:23:14,426] - [moving_topology_fts] [140513456174848] - INFO - Index building has begun...
[2016-08-18 17:23:14,468] - [moving_topology_fts] [140513456174848] - INFO - Index count for default_index_1: 7167
[2016-08-18 17:23:14,469] - [fts_base] [140513456174848] - INFO - Starting failover for nodes:[ip:172.23.105.190 port:8091 ssh_username:root] at C1 cluster 172.23.105.96
[2016-08-18 17:23:14,593] - [task] [140513447782144] - INFO - Failing over 172.23.105.190:8091 with graceful=True
[2016-08-18 17:23:14,603] - [rest_client] [140513447782144] - INFO - fail_over node ns_1@172.23.105.190 successful
[2016-08-18 17:23:14,603] - [task] [140513447782144] - INFO - 0 seconds sleep after failover, for nodes to go pending....
[2016-08-18 17:23:14,603] - [fts_base] [140513456174848] - INFO - sleep for 30 secs.  ...
[2016-08-18 17:23:44,671] - [rest_client] [140513456174848] - INFO - add_back_node ns_1@172.23.105.190 successful
[2016-08-18 17:23:44,671] - [rest_client] [140513456174848] - INFO - Going to set recoveryType=delta for node :: ns_1@172.23.105.190
[2016-08-18 17:23:44,684] - [rest_client] [140513456174848] - INFO - recoveryType for node ns_1@172.23.105.190 set successful
[2016-08-18 17:23:45,665] - [rest_client] [140513447782144] - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.105.96%2Cns_1%40172.23.105.190
[2016-08-18 17:23:45,701] - [rest_client] [140513447782144] - INFO - rebalance operation started
[2016-08-18 17:23:45,713] - [rest_client] [140513447782144] - INFO - rebalance percentage : 0.00 %
[2016-08-18 17:23:55,750] - [rest_client] [140513447782144] - INFO - rebalance percentage : 14.21 %
[2016-08-18 17:24:05,788] - [rest_client] [140513447782144] - INFO - rebalance percentage : 40.58 %
[2016-08-18 17:24:15,818] - [rest_client] [140513447782144] - INFO - rebalance percentage : 69.29 %
[2016-08-18 17:24:25,859] - [task] [140513447782144] - INFO - rebalancing was completed with progress: 100% in 40.1577289104 sec
[2016-08-18 17:24:25,860] - [fts_base] [140513456174848] - INFO - Validating index distribution for default_index_1 ...
[2016-08-18 17:24:25,919] - [fts_base] [140513456174848] - INFO - Validated: Number of PIndexes = 32
[2016-08-18 17:24:25,933] - [fts_base] [140513456174848] - INFO - Validated: Every pIndex serves 32 partitions or lesser
[2016-08-18 17:24:25,933] - [fts_base] [140513456174848] - INFO - Validated: pIndexes are distributed across [u'e6b2a25cc34d8ea4a349ad0981b179d6'] 
[2016-08-18 17:24:25,934] - [fts_base] [140513456174848] - INFO - Expecting num of partitions in each node in range 992-1024
[2016-08-18 17:24:25,934] - [fts_base] [140513456174848] - INFO - Validated: Node e6b2a25cc34d8ea4a349ad0981b179d6 houses 32 pindexes which serve 1024 partitions
[2016-08-18 17:24:26,042] - [fts_base] [140513456174848] - INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 22584
[2016-08-18 17:24:32,166] - [fts_base] [140513456174848] - INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 26207
[2016-08-18 17:24:38,276] - [fts_base] [140513456174848] - INFO - Docs in bucket = 30000, docs in FTS index 'default_index_1': 30000
[2016-08-18 17:24:38,276] - [fts_base] [140513456174848] - INFO - FTS indexed 30000 docs in 0.21 mins
[2016-08-18 17:24:38,374] - [fts_base] [140513456174848] - INFO - Docs in index default_index_1=30000, bucket docs=30000
[2016-08-18 17:24:38,385] - [rest_client] [140513456174848] - INFO - /diag/eval status on 172.23.105.96:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))).
[2016-08-18 17:24:38,387] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-08-18 17:24:38,631] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.96
[2016-08-18 17:24:39,220] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.96: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:39,327] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:39,328] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:39,329] - [fts_base] [140513456174848] - INFO - Initial 'panic:' count on 172.23.105.96 :0, now :0
[2016-08-18 17:24:39,329] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.96: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:39,435] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:39,436] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:39,436] - [fts_base] [140513456174848] - INFO - Initial 'crash in forestdb' count on 172.23.105.96 :0, now :0
[2016-08-18 17:24:39,554] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-08-18 17:24:39,833] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.190
[2016-08-18 17:24:40,375] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.190: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:41,185] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:41,187] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:41,187] - [fts_base] [140513456174848] - INFO - Initial 'panic:' count on 172.23.105.190 :0, now :0
[2016-08-18 17:24:41,187] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.190: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:41,827] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:41,828] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:41,828] - [fts_base] [140513456174848] - INFO - Initial 'crash in forestdb' count on 172.23.105.190 :0, now :0
[2016-08-18 17:24:41,944] - [remote_util] [140513456174848] - INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-08-18 17:24:42,188] - [remote_util] [140513456174848] - INFO - Connected to 172.23.106.66
[2016-08-18 17:24:42,729] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.106.66: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:42,851] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:42,853] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:42,853] - [fts_base] [140513456174848] - INFO - Initial 'panic:' count on 172.23.106.66 :0, now :0
[2016-08-18 17:24:42,853] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.106.66: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:42,969] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:42,970] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:42,970] - [fts_base] [140513456174848] - INFO - Initial 'crash in forestdb' count on 172.23.106.66 :0, now :0
[2016-08-18 17:24:43,087] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-08-18 17:24:43,331] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.69
[2016-08-18 17:24:43,856] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.69: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:43,973] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:43,976] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:43,976] - [fts_base] [140513456174848] - INFO - Initial 'panic:' count on 172.23.105.69 :0, now :0
[2016-08-18 17:24:43,977] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.69: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:44,088] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:44,089] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:44,090] - [fts_base] [140513456174848] - INFO - Initial 'crash in forestdb' count on 172.23.105.69 :0, now :0
[2016-08-18 17:24:44,208] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-08-18 17:24:44,442] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.54
[2016-08-18 17:24:45,034] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.54: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:45,186] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:45,187] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:45,188] - [fts_base] [140513456174848] - INFO - Initial 'panic:' count on 172.23.105.54 :0, now :0
[2016-08-18 17:24:45,189] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.54: zgrep "crash in forestdb" "/opt/couchbase/var/lib/couchbase/logs"/fts.log* | wc -l
[2016-08-18 17:24:45,327] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:45,328] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:45,329] - [fts_base] [140513456174848] - INFO - Initial 'crash in forestdb' count on 172.23.105.54 :0, now :0
[2016-08-18 17:24:45,444] - [fts_base] [140513456174848] - INFO - ====  FTSbasetests cleanup is started for test #15 graceful_failover_and_delta_recovery_during_index_building ====
[2016-08-18 17:24:45,469] - [fts_base] [140513456174848] - INFO - Deleting fulltext-index default_index_1 on 172.23.105.190
[2016-08-18 17:24:45,516] - [remote_util] [140513456174848] - INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-08-18 17:24:45,781] - [remote_util] [140513456174848] - INFO - Connected to 172.23.105.190
[2016-08-18 17:24:46,350] - [remote_util] [140513456174848] - INFO - running command.raw on 172.23.105.190: ls /opt/couchbase/var/lib/couchbase/data/@fts |grep default_index_1*.pindex | wc -l
[2016-08-18 17:24:46,444] - [remote_util] [140513456174848] - INFO - command executed successfully
[2016-08-18 17:24:46,444] - [fts_base] [140513456174848] - INFO - 0
[2016-08-18 17:24:46,446] - [fts_base] [140513456174848] - INFO - Validated: all index files for default_index_1 deleted from disk
[2016-08-18 17:24:46,446] - [fts_base] [140513456174848] - INFO - removing nodes from cluster ...
[2016-08-18 17:24:46,461] - [fts_base] [140513456174848] - INFO - cleanup [ip:172.23.105.96 port:8091 ssh_username:root, ip:172.23.105.190 port:8091 ssh_username:root]
[2016-08-18 17:24:46,488] - [bucket_helper] [140513456174848] - INFO - deleting existing buckets [u'default'] on 172.23.105.96
[2016-08-18 17:24:46,489] - [bucket_helper] [140513456174848] - INFO - remove bucket default ...
[2016-08-18 17:24:48,378] - [bucket_helper] [140513456174848] - INFO - deleted bucket : default from 172.23.105.96
[2016-08-18 17:24:48,378] - [bucket_helper] [140513456174848] - INFO - waiting for bucket deletion to complete....
[2016-08-18 17:24:48,383] - [rest_client] [140513456174848] - INFO - node 172.23.105.96 existing buckets : []
[2016-08-18 17:24:48,409] - [cluster_helper] [140513456174848] - INFO - rebalancing all nodes in order to remove nodes
[2016-08-18 17:24:48,413] - [rest_client] [140513456174848] - INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.105.190&user=Administrator&knownNodes=ns_1%40172.23.105.96%2Cns_1%40172.23.105.190
[2016-08-18 17:24:48,421] - [rest_client] [140513456174848] - INFO - rebalance operation started
[2016-08-18 17:24:48,427] - [rest_client] [140513456174848] - INFO - rebalance percentage : 0.00 %
[2016-08-18 17:24:58,445] - [rest_client] [140513456174848] - INFO - rebalance percentage : 75.00 %
[2016-08-18 17:25:18,477] - [rest_client] [140513456174848] - INFO - rebalance progress took 30.0556359291 seconds 
[2016-08-18 17:25:18,477] - [rest_client] [140513456174848] - INFO - sleep for 10 seconds after rebalance...
[2016-08-18 17:25:28,502] - [cluster_helper] [140513456174848] - INFO - removed all the nodes from cluster associated with ip:172.23.105.96 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.190', 8091)]
[2016-08-18 17:25:28,510] - [cluster_helper] [140513456174848] - INFO - waiting for ns_server @ 172.23.105.96:8091
[2016-08-18 17:25:28,516] - [cluster_helper] [140513456174848] - INFO - ns_server @ 172.23.105.96:8091 is running
[2016-08-18 17:25:28,525] - [bucket_helper] [140513456174848] - INFO - deleting existing buckets [] on 172.23.105.190
[2016-08-18 17:25:28,555] - [cluster_helper] [140513456174848] - INFO - waiting for ns_server @ 172.23.105.190:8091
[2016-08-18 17:25:28,566] - [cluster_helper] [140513456174848] - INFO - ns_server @ 172.23.105.190:8091 is running
[2016-08-18 17:25:28,567] - [fts_base] [140513456174848] - INFO - ====  FTSbasetests cleanup is finished for test #15 graceful_failover_and_delta_recovery_during_index_building ===
[2016-08-18 17:25:28,569] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-08-18 17:25:28,814] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.96
[2016-08-18 17:25:29,415] - [remote_util] [140513690150720] - INFO - 172.23.105.96 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:25:29,580] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-08-18 17:25:29,854] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.190
[2016-08-18 17:25:30,579] - [remote_util] [140513690150720] - INFO - 172.23.105.190 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:25:30,761] - [remote_util] [140513690150720] - INFO - found these files : ['/tmp//core.mc:writer_4.27638']
[2016-08-18 17:25:30,801] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.190: /opt/couchbase/bin/tools/cbanalyze-core /tmp//core.mc:writer_4.27638 -f /tmp/core-172.23.105.190-0.log
[2016-08-18 17:25:47,185] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:25:47,347] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.190: mkdir -p /tmp/backup_crash/18_08_2016_17_25;mv -f /tmp/core* /tmp/backup_crash/18_08_2016_17_25;mv -f /opt/couchbase/var/lib/couchbase/erl_crash.dump* /tmp/backup_crash/18_08_2016_17_25; mv -f /opt/couchbase/var/lib/couchbase/crash/*.dmp /tmp/backup_crash/18_08_2016_17_25;
[2016-08-18 17:25:47,396] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:25:47,397] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.190: ls -la /tmp/backup_crash/18_08_2016_17_25
[2016-08-18 17:25:47,494] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:25:47,499] - [remote_util] [140513690150720] - INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-08-18 17:25:47,782] - [remote_util] [140513690150720] - INFO - Connected to 172.23.106.66
[2016-08-18 17:25:48,382] - [remote_util] [140513690150720] - INFO - 172.23.106.66 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:25:48,602] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-08-18 17:25:48,852] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.69
[2016-08-18 17:25:49,536] - [remote_util] [140513690150720] - INFO - 172.23.105.69 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:25:49,698] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-08-18 17:25:49,919] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.54
[2016-08-18 17:25:50,566] - [remote_util] [140513690150720] - INFO - 172.23.105.54 **** The version file /opt/couchbase/ VERSION.txt  exists
[2016-08-18 17:27:54,609] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.96 with username : root password : couchbase ssh_key: 
[2016-08-18 17:27:54,849] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.96
[2016-08-18 17:27:55,427] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.96: /opt/couchbase/bin/cbcollect_info 172.23.105.96-20160818-1727-diag.zip
[2016-08-18 17:29:25,886] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:29:27,154] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.96: rm -f /root/172.23.105.96-20160818-1727-diag.zip
[2016-08-18 17:29:27,200] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:29:27,204] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.190 with username : root password : couchbase ssh_key: 
[2016-08-18 17:29:27,447] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.190
[2016-08-18 17:29:27,997] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.190: /opt/couchbase/bin/cbcollect_info 172.23.105.190-20160818-1729-diag.zip
[2016-08-18 17:30:47,140] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:30:48,132] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.190: rm -f /root/172.23.105.190-20160818-1729-diag.zip
[2016-08-18 17:30:48,178] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:30:48,296] - [remote_util] [140513690150720] - INFO - connecting to 172.23.106.66 with username : root password : couchbase ssh_key: 
[2016-08-18 17:30:48,538] - [remote_util] [140513690150720] - INFO - Connected to 172.23.106.66
[2016-08-18 17:30:49,123] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.106.66: /opt/couchbase/bin/cbcollect_info 172.23.106.66-20160818-1730-diag.zip
[2016-08-18 17:32:01,164] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:32:01,473] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.106.66: rm -f /root/172.23.106.66-20160818-1730-diag.zip
[2016-08-18 17:32:01,509] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:32:01,513] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.69 with username : root password : couchbase ssh_key: 
[2016-08-18 17:32:01,757] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.69
[2016-08-18 17:32:02,381] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.69: /opt/couchbase/bin/cbcollect_info 172.23.105.69-20160818-1732-diag.zip
[2016-08-18 17:33:15,011] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:33:15,292] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.69: rm -f /root/172.23.105.69-20160818-1732-diag.zip
[2016-08-18 17:33:15,322] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:33:15,327] - [remote_util] [140513690150720] - INFO - connecting to 172.23.105.54 with username : root password : couchbase ssh_key: 
[2016-08-18 17:33:15,548] - [remote_util] [140513690150720] - INFO - Connected to 172.23.105.54
[2016-08-18 17:33:16,175] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.54: /opt/couchbase/bin/cbcollect_info 172.23.105.54-20160818-1733-diag.zip
[2016-08-18 17:34:37,602] - [remote_util] [140513690150720] - INFO - command executed successfully
[2016-08-18 17:34:38,411] - [remote_util] [140513690150720] - INFO - running command.raw on 172.23.105.54: rm -f /root/172.23.105.54-20160818-1733-diag.zip
[2016-08-18 17:34:38,455] - [remote_util] [140513690150720] - INFO - command executed successfully