[2018-10-18 01:11:42,419] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:11:42,621] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.202 [2018-10-18 01:11:43,472] - [remote_util] [140554426750720] - INFO - 172.23.106.202 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:11:43,682] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:11:43,814] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.204 [2018-10-18 01:11:44,462] - [remote_util] [140554426750720] - INFO - 172.23.106.204 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:11:44,611] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:11:44,707] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.208 [2018-10-18 01:11:44,952] - [remote_util] [140554426750720] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:11:45,427] - [remote_util] [140554426750720] - INFO - 172.23.106.208 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:11:45,486] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:11:45,651] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.209 [2018-10-18 01:11:46,254] - [remote_util] [140554426750720] - INFO - 172.23.106.209 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:11:46,420] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:11:46,585] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.63 [2018-10-18 01:11:47,238] - [remote_util] [140554426750720] - INFO - 172.23.106.63 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:11:47,363] - [remote_util] [140554426750720] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:11:47,460] - [remote_util] [140554426750720] - INFO - Connected to 172.23.98.14 [2018-10-18 01:11:47,731] - [remote_util] [140554426750720] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:11:48,177] - [remote_util] [140554426750720] - INFO - 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:11:48,300] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:11:48,502] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:11:49,400] - [rest_client] [140554369615616] - INFO - Node version in cluster 6.5.0-1441-enterprise [2018-10-18 01:11:49,423] - [rest_client] [140554369615616] - INFO - Node versions in cluster [u'6.5.0-1441-enterprise', u'6.5.0-1441-enterprise', u'6.5.0-1441-enterprise', u'6.5.0-1441-enterprise'] [2018-10-18 01:11:49,424] - [basetestcase] [140554369615616] - INFO - ============== basetestcase setup was started for test #16 online_cluster_upgrade============== [2018-10-18 01:11:49,512] - [bucket_helper] [140554369615616] - INFO - deleting existing buckets [u'default', u'sasl_bucket_1', u'standard_bucket_1'] on 172.23.106.202 [2018-10-18 01:11:49,512] - [bucket_helper] [140554369615616] - INFO - remove bucket default ... [2018-10-18 01:11:50,677] - [bucket_helper] [140554369615616] - INFO - deleted bucket : default from 172.23.106.202 [2018-10-18 01:11:50,677] - [bucket_helper] [140554369615616] - INFO - waiting for bucket deletion to complete.... [2018-10-18 01:11:50,709] - [rest_client] [140554369615616] - INFO - node 172.23.106.202 existing buckets : [u'sasl_bucket_1', u'standard_bucket_1'] [2018-10-18 01:11:50,709] - [bucket_helper] [140554369615616] - INFO - remove bucket sasl_bucket_1 ... [2018-10-18 01:11:52,151] - [bucket_helper] [140554369615616] - INFO - deleted bucket : sasl_bucket_1 from 172.23.106.202 [2018-10-18 01:11:52,151] - [bucket_helper] [140554369615616] - INFO - waiting for bucket deletion to complete.... [2018-10-18 01:11:52,169] - [rest_client] [140554369615616] - INFO - node 172.23.106.202 existing buckets : [u'standard_bucket_1'] [2018-10-18 01:11:52,170] - [bucket_helper] [140554369615616] - INFO - remove bucket standard_bucket_1 ... [2018-10-18 01:11:52,991] - [bucket_helper] [140554369615616] - INFO - deleted bucket : standard_bucket_1 from 172.23.106.202 [2018-10-18 01:11:52,992] - [bucket_helper] [140554369615616] - INFO - waiting for bucket deletion to complete.... [2018-10-18 01:11:52,995] - [rest_client] [140554369615616] - INFO - node 172.23.106.202 existing buckets : [] [2018-10-18 01:11:52,995] - [bucket_helper] [140554369615616] - INFO - sleep 2 seconds to make sure all buckets ([u'default', u'sasl_bucket_1', u'standard_bucket_1']) were deleted completely. [2018-10-18 01:11:55,040] - [bucket_helper] [140554369615616] - INFO - deleting existing buckets [u'default', u'sasl_bucket_1', u'standard_bucket_1'] on 172.23.106.208 [2018-10-18 01:11:55,041] - [bucket_helper] [140554369615616] - INFO - remove bucket default ... [2018-10-18 01:11:55,844] - [bucket_helper] [140554369615616] - INFO - deleted bucket : default from 172.23.106.208 [2018-10-18 01:11:55,844] - [bucket_helper] [140554369615616] - INFO - waiting for bucket deletion to complete.... [2018-10-18 01:11:55,864] - [rest_client] [140554369615616] - INFO - node 172.23.106.208 existing buckets : [u'sasl_bucket_1', u'standard_bucket_1'] [2018-10-18 01:11:55,865] - [bucket_helper] [140554369615616] - INFO - remove bucket sasl_bucket_1 ... [2018-10-18 01:11:57,144] - [bucket_helper] [140554369615616] - INFO - deleted bucket : sasl_bucket_1 from 172.23.106.208 [2018-10-18 01:11:57,145] - [bucket_helper] [140554369615616] - INFO - waiting for bucket deletion to complete.... [2018-10-18 01:11:57,156] - [rest_client] [140554369615616] - INFO - node 172.23.106.208 existing buckets : [u'standard_bucket_1'] [2018-10-18 01:11:57,156] - [bucket_helper] [140554369615616] - INFO - remove bucket standard_bucket_1 ... [2018-10-18 01:11:58,042] - [bucket_helper] [140554369615616] - INFO - deleted bucket : standard_bucket_1 from 172.23.106.208 [2018-10-18 01:11:58,042] - [bucket_helper] [140554369615616] - INFO - waiting for bucket deletion to complete.... [2018-10-18 01:11:58,045] - [rest_client] [140554369615616] - INFO - node 172.23.106.208 existing buckets : [] [2018-10-18 01:11:58,045] - [bucket_helper] [140554369615616] - INFO - sleep 2 seconds to make sure all buckets ([u'default', u'sasl_bucket_1', u'standard_bucket_1']) were deleted completely. [2018-10-18 01:12:00,116] - [cluster_helper] [140554369615616] - INFO - rebalancing all nodes in order to remove nodes [2018-10-18 01:12:00,122] - [rest_client] [140554369615616] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.106.204,ns_1@172.23.98.14,ns_1@172.23.106.63', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.204,ns_1@172.23.98.14,ns_1@172.23.106.202,ns_1@172.23.106.63'} [2018-10-18 01:12:00,130] - [rest_client] [140554369615616] - INFO - rebalance operation started [2018-10-18 01:12:00,138] - [rest_client] [140554369615616] - INFO - rebalance percentage : 0.00 % [2018-10-18 01:12:20,166] - [rest_client] [140554369615616] - INFO - rebalance progress took 20.04 seconds [2018-10-18 01:12:20,166] - [rest_client] [140554369615616] - INFO - sleep for 10 seconds after rebalance... [2018-10-18 01:12:30,205] - [cluster_helper] [140554369615616] - INFO - removed all the nodes from cluster associated with ip:172.23.106.202 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.204', 8091), (u'ns_1@172.23.98.14', 8091), (u'ns_1@172.23.106.63', 8091)] [2018-10-18 01:12:30,212] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.202:8091 [2018-10-18 01:12:30,219] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.202:8091 is running [2018-10-18 01:12:30,223] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.204:8091 [2018-10-18 01:12:30,226] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.204:8091 is running [2018-10-18 01:12:30,231] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.208:8091 [2018-10-18 01:12:30,235] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.208:8091 is running [2018-10-18 01:12:30,241] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.209:8091 [2018-10-18 01:12:30,245] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.209:8091 is running [2018-10-18 01:12:30,251] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.63:8091 [2018-10-18 01:12:30,259] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.63:8091 is running [2018-10-18 01:12:30,264] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.98.14:8091 [2018-10-18 01:12:30,269] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.98.14:8091 is running [2018-10-18 01:12:30,270] - [basetestcase] [140554369615616] - INFO - any cluster operation in setup will be skipped [2018-10-18 01:12:30,270] - [basetestcase] [140554369615616] - INFO - ============== basetestcase setup was finished for test #16 online_cluster_upgrade ============== [2018-10-18 01:12:30,272] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:12:30,474] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:12:45,062] - [basetestcase] [140554369615616] - INFO - cannot find service node n1ql in cluster [2018-10-18 01:12:45,064] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:12:45,266] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:12:46,063] - [newupgradebasetest] [140554369615616] - INFO - will install 4.6.3-4136 on ['172.23.106.202', '172.23.106.204', '172.23.106.208', '172.23.106.209'] [2018-10-18 01:12:46,070] - [remote_util] [140543786632960] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:12:46,072] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:12:46,074] - [remote_util] [140543778240256] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:12:46,076] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:12:46,285] - [remote_util] [140543786632960] - INFO - Connected to 172.23.106.204 [2018-10-18 01:12:46,312] - [remote_util] [140543778240256] - INFO - Connected to 172.23.106.208 [2018-10-18 01:12:46,365] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.209 [2018-10-18 01:12:46,404] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.202 [2018-10-18 01:12:46,554] - [remote_util] [140543778240256] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:12:46,785] - [remote_util] [140543786632960] - INFO - 172.23.106.204 *****In couchbase uninstall**** [2018-10-18 01:12:46,786] - [remote_util] [140543786632960] - INFO - CentOS [2018-10-18 01:12:46,869] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: killall -9 rpm [2018-10-18 01:12:46,889] - [remote_util] [140543769847552] - INFO - 172.23.106.209 *****In couchbase uninstall**** [2018-10-18 01:12:46,889] - [remote_util] [140543769847552] - INFO - CentOS [2018-10-18 01:12:46,904] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:46,904] - [remote_util] [140543786632960] - ERROR - rpm: no process killed [2018-10-18 01:12:46,904] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: rm -f /var/lib/rpm/.rpm.lock [2018-10-18 01:12:46,953] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:46,955] - [remote_util] [140543778240256] - INFO - 172.23.106.208 *****In couchbase uninstall**** [2018-10-18 01:12:46,955] - [remote_util] [140543778240256] - INFO - CentOS [2018-10-18 01:12:47,013] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: killall -9 rpm [2018-10-18 01:12:47,047] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:47,047] - [remote_util] [140543769847552] - ERROR - rpm: no process killed [2018-10-18 01:12:47,048] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: rm -f /var/lib/rpm/.rpm.lock [2018-10-18 01:12:47,098] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:47,123] - [remote_util] [140543811811072] - INFO - 172.23.106.202 *****In couchbase uninstall**** [2018-10-18 01:12:47,123] - [remote_util] [140543811811072] - INFO - CentOS [2018-10-18 01:12:47,128] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: killall -9 rpm [2018-10-18 01:12:47,134] - [remote_util] [140543786632960] - INFO - running rpm -e to remove couchbase-server [2018-10-18 01:12:47,134] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: rpm -e couchbase-server [2018-10-18 01:12:47,163] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:47,163] - [remote_util] [140543778240256] - ERROR - rpm: no process found [2018-10-18 01:12:47,163] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: rm -f /var/lib/rpm/.rpm.lock [2018-10-18 01:12:47,235] - [remote_util] [140543769847552] - INFO - running rpm -e to remove couchbase-server [2018-10-18 01:12:47,235] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: rpm -e couchbase-server [2018-10-18 01:12:47,244] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:47,274] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: killall -9 rpm [2018-10-18 01:12:47,349] - [remote_util] [140543778240256] - INFO - running rpm -e to remove couchbase-server [2018-10-18 01:12:47,349] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: rpm -e couchbase-server [2018-10-18 01:12:47,391] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:47,392] - [remote_util] [140543811811072] - ERROR - rpm: no process killed [2018-10-18 01:12:47,392] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: rm -f /var/lib/rpm/.rpm.lock [2018-10-18 01:12:47,426] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:47,545] - [remote_util] [140543811811072] - INFO - running rpm -e to remove couchbase-server [2018-10-18 01:12:47,545] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: rpm -e couchbase-server [2018-10-18 01:12:49,219] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:49,219] - [remote_util] [140543778240256] - ERROR - **** Uninstalling on server: 172.23.106.208 **** [2018-10-18 01:12:49,219] - [remote_util] [140543778240256] - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory [2018-10-18 01:12:49,219] - [remote_util] [140543778240256] - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave [2018-10-18 01:12:49,219] - [remote_util] [140543778240256] - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave [2018-10-18 01:12:49,220] - [remote_util] [140543778240256] - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave [2018-10-18 01:12:49,220] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: pkill -u couchbase [2018-10-18 01:12:49,238] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:49,238] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: ps -ef | grep couchbase [2018-10-18 01:12:49,335] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:49,335] - [remote_util] [140543778240256] - INFO - root 476 317 0 01:12 ? 00:00:00 bash -c ps -ef | grep couchbase [2018-10-18 01:12:49,335] - [remote_util] [140543778240256] - INFO - root 482 476 0 01:12 ? 00:00:00 grep couchbase [2018-10-18 01:12:50,284] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:50,285] - [remote_util] [140543769847552] - ERROR - **** Uninstalling on server: 172.23.106.209 **** [2018-10-18 01:12:50,285] - [remote_util] [140543769847552] - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave [2018-10-18 01:12:50,285] - [remote_util] [140543769847552] - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave [2018-10-18 01:12:50,286] - [remote_util] [140543769847552] - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave [2018-10-18 01:12:50,287] - [remote_util] [140543769847552] - INFO - **** Uninstalling on server: 172.23.106.209 **** [2018-10-18 01:12:50,288] - [remote_util] [140543769847552] - INFO - Stopping couchbase-server [2018-10-18 01:12:50,289] - [remote_util] [140543769847552] - INFO - [ OK ] [2018-10-18 01:12:50,290] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: pkill -u couchbase [2018-10-18 01:12:50,310] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:50,310] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: ps -ef | grep couchbase [2018-10-18 01:12:50,408] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:50,408] - [remote_util] [140543769847552] - INFO - root 4841 4732 0 01:12 ? 00:00:00 bash -c ps -ef | grep couchbase [2018-10-18 01:12:50,408] - [remote_util] [140543769847552] - INFO - root 4845 4841 0 01:12 ? 00:00:00 grep couchbase [2018-10-18 01:12:50,683] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:50,683] - [remote_util] [140543786632960] - ERROR - **** Uninstalling on server: 172.23.106.204 **** [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - INFO - **** Uninstalling on server: 172.23.106.204 **** [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - INFO - Stopping couchbase-server [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - INFO - [ OK ] [2018-10-18 01:12:50,684] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: pkill -u couchbase [2018-10-18 01:12:50,704] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:50,704] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: ps -ef | grep couchbase [2018-10-18 01:12:50,801] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:50,801] - [remote_util] [140543786632960] - INFO - root 22249 22092 0 01:12 ? 00:00:00 bash -c ps -ef | grep couchbase [2018-10-18 01:12:50,801] - [remote_util] [140543786632960] - INFO - root 22253 22249 0 01:12 ? 00:00:00 grep couchbase [2018-10-18 01:12:50,864] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: kill -9 $(ps aux | pgrep 'memcached') [2018-10-18 01:12:50,944] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:50,945] - [remote_util] [140543778240256] - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] [2018-10-18 01:12:50,945] - [remote_util] [140543778240256] - INFO - running command.raw on 172.23.106.208: ipcrm [2018-10-18 01:12:51,017] - [remote_util] [140543778240256] - INFO - command executed successfully [2018-10-18 01:12:52,147] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:52,147] - [remote_util] [140543811811072] - ERROR - **** Uninstalling on server: 172.23.106.202 **** [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip_start saved as /opt/couchbase/var/lib/couchbase/ip_start.rpmsave [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - ERROR - warning: erase unlink of /opt/couchbase/var/lib/couchbase/ip failed: No such file or directory [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - INFO - **** Uninstalling on server: 172.23.106.202 **** [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - INFO - Stopping couchbase-server [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - INFO - [ OK ] [2018-10-18 01:12:52,148] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: pkill -u couchbase [2018-10-18 01:12:52,215] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:52,215] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: ps -ef | grep couchbase [2018-10-18 01:12:52,345] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:52,345] - [remote_util] [140543811811072] - INFO - root 1605 1351 0 01:12 ? 00:00:00 bash -c ps -ef | grep couchbase [2018-10-18 01:12:52,345] - [remote_util] [140543811811072] - INFO - root 1613 1605 0 01:12 ? 00:00:00 grep couchbase [2018-10-18 01:12:52,424] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: kill -9 $(ps aux | pgrep 'memcached') [2018-10-18 01:12:52,521] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:52,522] - [remote_util] [140543786632960] - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] [2018-10-18 01:12:52,522] - [remote_util] [140543786632960] - INFO - running command.raw on 172.23.106.204: ipcrm [2018-10-18 01:12:52,541] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: kill -9 $(ps aux | pgrep 'memcached') [2018-10-18 01:12:52,570] - [remote_util] [140543786632960] - INFO - command executed successfully [2018-10-18 01:12:52,639] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:52,639] - [remote_util] [140543769847552] - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] [2018-10-18 01:12:52,639] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.209: ipcrm [2018-10-18 01:12:52,688] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:12:54,411] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: kill -9 $(ps aux | pgrep 'memcached') [2018-10-18 01:12:54,509] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:54,509] - [remote_util] [140543811811072] - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] [2018-10-18 01:12:54,509] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: ipcrm [2018-10-18 01:12:54,558] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:12:54,673] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:12:54,875] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:12:55,814] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:12:55,947] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:12:56,606] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:12:56,704] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:12:56,948] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:12:57,527] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:12:57,692] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:12:58,393] - [install] [140543811811072] - INFO - ********CouchbaseServerInstaller:install [2018-10-18 01:12:58,393] - [install] [140543275300608] - INFO - ********CouchbaseServerInstaller:install [2018-10-18 01:12:58,393] - [install] [140543769847552] - INFO - ********CouchbaseServerInstaller:install [2018-10-18 01:12:58,395] - [install] [140543266907904] - INFO - ********CouchbaseServerInstaller:install [2018-10-18 01:12:58,398] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:12:58,401] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:12:58,401] - [remote_util] [140543275300608] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:12:58,403] - [remote_util] [140543266907904] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:12:58,620] - [remote_util] [140543275300608] - INFO - Connected to 172.23.106.204 [2018-10-18 01:12:58,627] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.208 [2018-10-18 01:12:58,653] - [remote_util] [140543266907904] - INFO - Connected to 172.23.106.209 [2018-10-18 01:12:58,728] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.202 [2018-10-18 01:12:58,840] - [remote_util] [140543769847552] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:12:59,186] - [remote_util] [140543275300608] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:12:59,231] - [remote_util] [140543266907904] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:12:59,360] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:12:59,378] - [remote_util] [140543275300608] - INFO - Connected to 172.23.106.204 [2018-10-18 01:12:59,431] - [remote_util] [140543266907904] - INFO - Connected to 172.23.106.209 [2018-10-18 01:12:59,441] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.208 [2018-10-18 01:12:59,584] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:12:59,705] - [remote_util] [140543769847552] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:12:59,785] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.202 [2018-10-18 01:12:59,901] - [remote_util] [140543275300608] - INFO - Check if url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm is ok [2018-10-18 01:12:59,987] - [remote_util] [140543275300608] - INFO - This url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm is live [2018-10-18 01:13:00,053] - [remote_util] [140543275300608] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:13:00,134] - [remote_util] [140543266907904] - INFO - Check if url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm is ok [2018-10-18 01:13:00,138] - [remote_util] [140543266907904] - INFO - This url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm is live [2018-10-18 01:13:00,144] - [remote_util] [140543769847552] - INFO - Check if url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos7.x86_64.rpm is ok [2018-10-18 01:13:00,185] - [remote_util] [140543769847552] - INFO - This url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos7.x86_64.rpm is live [2018-10-18 01:13:00,218] - [remote_util] [140543275300608] - INFO - Connected to 172.23.106.204 [2018-10-18 01:13:00,253] - [remote_util] [140543266907904] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:13:00,337] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:13:00,420] - [remote_util] [140543266907904] - INFO - Connected to 172.23.106.209 [2018-10-18 01:13:00,434] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.208 [2018-10-18 01:13:00,632] - [remote_util] [140543811811072] - INFO - Check if url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm is ok [2018-10-18 01:13:00,635] - [remote_util] [140543811811072] - INFO - This url http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm is live [2018-10-18 01:13:00,670] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:13:00,670] - [remote_util] [140543769847552] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:13:00,777] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: /sbin/iptables -F [2018-10-18 01:13:00,858] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:00,858] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: /sbin/iptables -t nat -F [2018-10-18 01:13:00,873] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.202 [2018-10-18 01:13:00,939] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:00,939] - [remote_util] [140543275300608] - INFO - Connect to node: 172.23.106.204 as user: root [2018-10-18 01:13:00,948] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: /sbin/iptables -F [2018-10-18 01:13:01,041] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:01,041] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: /sbin/iptables -t nat -F [2018-10-18 01:13:01,072] - [remote_util] [140543275300608] - INFO - Connected to 172.23.106.204 as root [2018-10-18 01:13:01,072] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_4.6.3-4136.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D [2018-10-18 01:13:01,106] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: /sbin/iptables -F [2018-10-18 01:13:01,123] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:01,123] - [remote_util] [140543266907904] - INFO - Connect to node: 172.23.106.209 as user: root [2018-10-18 01:13:01,211] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:01,211] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: /sbin/iptables -t nat -F [2018-10-18 01:13:01,214] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:01,214] - [remote_util] [140543275300608] - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_4.6.3-4136.rpm': No such file or directory [2018-10-18 01:13:01,214] - [remote_util] [140543275300608] - ERROR - mv: cannot stat `core.*': No such file or directory [2018-10-18 01:13:01,214] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `backup_crash': Is a directory [2018-10-18 01:13:01,214] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `cb_bY9q': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `d1': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `d2': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `d3': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `d4': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `d5': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `d6': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `entbackup': Is a directory [2018-10-18 01:13:01,215] - [remote_util] [140543275300608] - ERROR - rm: cannot remove `export172.23.106.204': Is a directory [2018-10-18 01:13:01,216] - [remote_util] [140543275300608] - ERROR - mv: cannot stat `cb_bY9q/*': No such file or directory [2018-10-18 01:13:01,216] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_4.6.3-4136.rpm http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm;cd /tmp;ls -lh [2018-10-18 01:13:01,285] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:01,285] - [remote_util] [140543769847552] - INFO - Connect to node: 172.23.106.208 as user: root [2018-10-18 01:13:01,289] - [remote_util] [140543266907904] - INFO - Connected to 172.23.106.209 as root [2018-10-18 01:13:01,290] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_4.6.3-4136.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D [2018-10-18 01:13:01,383] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.208 as root [2018-10-18 01:13:01,383] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_4.6.3-4136.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D [2018-10-18 01:13:01,407] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:01,407] - [remote_util] [140543266907904] - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_4.6.3-4136.rpm': No such file or directory [2018-10-18 01:13:01,407] - [remote_util] [140543266907904] - ERROR - mv: cannot stat `core.*': No such file or directory [2018-10-18 01:13:01,407] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `backup_crash': Is a directory [2018-10-18 01:13:01,407] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `backups': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `cb_RRco': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `d1': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `d2': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `d3': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `d4': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `d5': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `d6': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `entbackup': Is a directory [2018-10-18 01:13:01,408] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `entbackup172.23.106.147': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `export172.23.106.209': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `hsperfdata_couchbase': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `measure-sched-delays': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmp3nitoP': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmp3So9xT': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmp4CN85c': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmp6uoVby': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmp8bXO2s': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpas4TEP': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpb70Bj6': Is a directory [2018-10-18 01:13:01,409] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpCfx97m': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpci4rM1': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpEAeh9l': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpFf4HPq': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpGn9Lin': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmp_hAzdn': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpJqLofw': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpkQeW0K': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmplKASrm': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpl_TYE0': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpM3mFg9': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpMX630e': Is a directory [2018-10-18 01:13:01,410] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmppaWkMl': Is a directory [2018-10-18 01:13:01,411] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmptHio7T': Is a directory [2018-10-18 01:13:01,411] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpY1pp9m': Is a directory [2018-10-18 01:13:01,411] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpZNutDY': Is a directory [2018-10-18 01:13:01,411] - [remote_util] [140543266907904] - ERROR - rm: cannot remove `tmpzsMlnK': Is a directory [2018-10-18 01:13:01,411] - [remote_util] [140543266907904] - ERROR - mv: cannot stat `cb_RRco/*': No such file or directory [2018-10-18 01:13:01,411] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_4.6.3-4136.rpm http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm;cd /tmp;ls -lh [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - mv: cannot stat ‘couchbase-server-enterprise_x86_64_4.6.3-4136.rpm’: No such file or directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - mv: cannot stat ‘core.*’: No such file or directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - rm: cannot remove ‘cb_HeiM’: Is a directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - rm: cannot remove ‘entbackup’: Is a directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - rm: cannot remove ‘systemd-private-74b195a8fd064c95a40edda950e5faab-ntpd.service-N4gJfJ’: Is a directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - ERROR - mv: cannot stat ‘cb_HeiM/*’: No such file or directory [2018-10-18 01:13:01,584] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_4.6.3-4136.rpm http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos7.x86_64.rpm;cd /tmp;ls -lh [2018-10-18 01:13:01,700] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: /sbin/iptables -F [2018-10-18 01:13:01,765] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:01,765] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: /sbin/iptables -t nat -F [2018-10-18 01:13:01,831] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:01,831] - [remote_util] [140543811811072] - INFO - Connect to node: 172.23.106.202 as user: root [2018-10-18 01:13:02,033] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.202 as root [2018-10-18 01:13:02,033] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise_x86_64_4.6.3-4136.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D [2018-10-18 01:13:02,332] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:02,332] - [remote_util] [140543811811072] - ERROR - mv: cannot stat `couchbase-server-enterprise_x86_64_4.6.3-4136.rpm': No such file or directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - mv: cannot stat `core.*': No such file or directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `cb_yAHR': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `export172.23.106.202': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `hsperfdata_root': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-3TN7AQ': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-HTD2Qa': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-I7Wt4t': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-Jo6O0u': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-muu5DO': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-otmZ6Z': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-qEKpPx': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-r7OdN7': Is a directory [2018-10-18 01:13:02,333] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `keyring-YMoDU3': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `orbit-arun': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `orbit-root': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `pulse-9wD1Sk1ULha2': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `pulse-SQaTPN5Mh0UI': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `ssh-GmjTmu1911': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `ssh-kmdEFA1749': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - rm: cannot remove `virtual-arun.y9GpOQ': Is a directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - ERROR - mv: cannot stat `cb_yAHR/*': No such file or directory [2018-10-18 01:13:02,334] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: cd /tmp;wget -q -O couchbase-server-enterprise_x86_64_4.6.3-4136.rpm http://172.23.120.24/builds/releases/4.6.3/couchbase-server-enterprise-4.6.3-centos6.x86_64.rpm;cd /tmp;ls -lh [2018-10-18 01:13:03,731] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - total 126M [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxr-xr-x 20 root root 4.0K Sep 30 07:54 backup_crash [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - -rw-r--r-- 1 root root 126M Aug 7 2017 couchbase-server-enterprise_x86_64_4.6.3-4136.rpm [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 03:56 d1 [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 03:56 d2 [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 03:56 d3 [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 03:56 d4 [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 03:56 d5 [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 03:56 d6 [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxrwx--- 4 root root 4.0K Jun 23 03:03 entbackup [2018-10-18 01:13:03,732] - [remote_util] [140543275300608] - INFO - drwxr-xr-x 2 root root 4.0K Jul 21 10:29 export172.23.106.204 [2018-10-18 01:13:03,757] - [remote_util] [140543275300608] - INFO - *****install server *** [2018-10-18 01:13:03,757] - [remote_util] [140543275300608] - INFO - deliverable_type : rpm [2018-10-18 01:13:03,757] - [remote_util] [140543275300608] - INFO - /tmp/couchbase-server-enterprise_x86_64_4.6.3-4136.rpm or /tmp/couchbase-server-enterprise [2018-10-18 01:13:03,757] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: /sbin/sysctl vm.swappiness=0 [2018-10-18 01:13:03,839] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:03,839] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: cat /etc/redhat-release [2018-10-18 01:13:03,920] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:03,920] - [remote_util] [140543275300608] - INFO - CentOS release 6.5 (Final) [2018-10-18 01:13:03,920] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: rpm -qa | grep pkgconfig [2018-10-18 01:13:04,250] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:04,250] - [remote_util] [140543275300608] - INFO - pkgconfig-0.23-9.1.el6.x86_64 [2018-10-18 01:13:05,835] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:05,835] - [remote_util] [140543769847552] - INFO - total 108M [2018-10-18 01:13:05,836] - [remote_util] [140543769847552] - INFO - -rw-r--r--. 1 root root 108M Aug 7 2017 couchbase-server-enterprise_x86_64_4.6.3-4136.rpm [2018-10-18 01:13:05,836] - [remote_util] [140543769847552] - INFO - drwxrwx---. 4 root root 44 Oct 12 18:34 entbackup [2018-10-18 01:13:05,836] - [remote_util] [140543769847552] - INFO - drwxr-x---. 2 couchbase couchbase 6 Oct 11 15:32 hsperfdata_couchbase [2018-10-18 01:13:05,836] - [remote_util] [140543769847552] - INFO - drwx------. 3 root root 16 Oct 17 11:29 systemd-private-74b195a8fd064c95a40edda950e5faab-ntpd.service-N4gJfJ [2018-10-18 01:13:05,850] - [remote_util] [140543769847552] - INFO - *****install server *** [2018-10-18 01:13:05,850] - [remote_util] [140543769847552] - INFO - deliverable_type : rpm [2018-10-18 01:13:05,851] - [remote_util] [140543769847552] - INFO - /tmp/couchbase-server-enterprise_x86_64_4.6.3-4136.rpm or /tmp/couchbase-server-enterprise [2018-10-18 01:13:05,851] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: /sbin/sysctl vm.swappiness=0 [2018-10-18 01:13:05,931] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:05,931] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: cat /etc/redhat-release [2018-10-18 01:13:06,012] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:06,012] - [remote_util] [140543769847552] - INFO - CentOS Linux release 7.3.1611 (Core) [2018-10-18 01:13:06,012] - [remote_util] [140543769847552] - INFO - no need to install pkgconfig [2018-10-18 01:13:06,238] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - total 126M [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxr-xr-x 1002 root root 36K Sep 23 22:00 backup_crash [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxrwx--- 4 root root 4.0K Mar 7 2018 backups [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - -rw-r--r-- 1 root root 126M Aug 7 2017 couchbase-server-enterprise_x86_64_4.6.3-4136.rpm [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 14:15 d1 [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 14:15 d2 [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 14:15 d3 [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 14:15 d4 [2018-10-18 01:13:06,239] - [remote_util] [140543266907904] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 14:15 d5 [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwxrwx--- 3 couchbase couchbase 4.0K Jun 15 14:15 d6 [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwxr-x--- 4 root root 4.0K Sep 28 14:06 entbackup [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwxrwx--- 4 root root 4.0K Nov 10 2017 entbackup172.23.106.147 [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwxr-xr-x 2 root root 4.0K Apr 21 18:51 export172.23.106.209 [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwxr-x--- 2 couchbase couchbase 4.0K Oct 11 15:32 hsperfdata_couchbase [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwxrwxr-x 3 1000 1000 4.0K Jan 23 2015 measure-sched-delays [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 4 2016 tmp3nitoP [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 3 2016 tmp3So9xT [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 27 2016 tmp4CN85c [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 3 2016 tmp6uoVby [2018-10-18 01:13:06,240] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 13 2016 tmp8bXO2s [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 1 2016 tmpas4TEP [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 4 2016 tmpb70Bj6 [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 3 2016 tmpCfx97m [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 29 2016 tmpci4rM1 [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmpEAeh9l [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 29 2016 tmpFf4HPq [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Jun 6 2017 tmpGn9Lin [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 3 2016 tmp_hAzdn [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Dec 8 2015 tmpJqLofw [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 27 2016 tmpkQeW0K [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 3 2016 tmplKASrm [2018-10-18 01:13:06,241] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 28 2016 tmpl_TYE0 [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 1 2016 tmpM3mFg9 [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmpMX630e [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K May 3 2016 tmppaWkMl [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 29 2016 tmptHio7T [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 4 2018 tmpY1pp9m [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 29 2016 tmpZNutDY [2018-10-18 01:13:06,242] - [remote_util] [140543266907904] - INFO - drwx------ 2 root root 4.0K Apr 29 2016 tmpzsMlnK [2018-10-18 01:13:06,333] - [remote_util] [140543266907904] - INFO - *****install server *** [2018-10-18 01:13:06,334] - [remote_util] [140543266907904] - INFO - deliverable_type : rpm [2018-10-18 01:13:06,334] - [remote_util] [140543266907904] - INFO - /tmp/couchbase-server-enterprise_x86_64_4.6.3-4136.rpm or /tmp/couchbase-server-enterprise [2018-10-18 01:13:06,334] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: /sbin/sysctl vm.swappiness=0 [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - total 126M [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - -rw-r--r--. 1 root root 126M Aug 7 2017 couchbase-server-enterprise_x86_64_4.6.3-4136.rpm [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwxr-xr-x. 2 root root 4.0K Oct 12 18:37 export172.23.106.202 [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwxr-xr-x. 2 root root 4.0K Oct 17 15:09 hsperfdata_root [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Dec 5 2016 keyring-3TN7AQ [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Nov 25 2017 keyring-HTD2Qa [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Oct 17 15:09 keyring-I7Wt4t [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Nov 25 2017 keyring-Jo6O0u [2018-10-18 01:13:06,365] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Dec 26 2017 keyring-muu5DO [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Dec 5 2016 keyring-otmZ6Z [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Oct 17 15:09 keyring-qEKpPx [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Dec 26 2017 keyring-r7OdN7 [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Dec 5 2016 keyring-YMoDU3 [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Oct 17 15:09 orbit-arun [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Oct 17 15:09 orbit-root [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Nov 19 2015 pulse-9wD1Sk1ULha2 [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Oct 17 15:09 pulse-SQaTPN5Mh0UI [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 root root 4.0K Oct 17 15:09 ssh-GmjTmu1911 [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Oct 17 15:09 ssh-kmdEFA1749 [2018-10-18 01:13:06,366] - [remote_util] [140543811811072] - INFO - drwx------. 2 arun arun 4.0K Oct 17 15:09 virtual-arun.y9GpOQ [2018-10-18 01:13:06,416] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:06,416] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: cat /etc/redhat-release [2018-10-18 01:13:06,432] - [remote_util] [140543811811072] - INFO - *****install server *** [2018-10-18 01:13:06,433] - [remote_util] [140543811811072] - INFO - deliverable_type : rpm [2018-10-18 01:13:06,433] - [remote_util] [140543811811072] - INFO - /tmp/couchbase-server-enterprise_x86_64_4.6.3-4136.rpm or /tmp/couchbase-server-enterprise [2018-10-18 01:13:06,433] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: /sbin/sysctl vm.swappiness=0 [2018-10-18 01:13:06,497] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:06,497] - [remote_util] [140543266907904] - INFO - CentOS release 6.4 (Final) [2018-10-18 01:13:06,497] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: rpm -qa | grep pkgconfig [2018-10-18 01:13:06,530] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:06,530] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: cat /etc/redhat-release [2018-10-18 01:13:06,596] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:06,596] - [remote_util] [140543811811072] - INFO - CentOS release 6.8 (Final) [2018-10-18 01:13:06,597] - [remote_util] [140543811811072] - INFO - no need to install pkgconfig [2018-10-18 01:13:06,828] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:06,829] - [remote_util] [140543266907904] - INFO - pkgconfig-0.23-9.1.el6.x86_64 [2018-10-18 01:13:08,334] - [remote_util] [140543275300608] - ERROR - Directory at /tmp DOES exist. Fx returns True [2018-10-18 01:13:08,433] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server [2018-10-18 01:13:08,515] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:08,626] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.106.204: /etc/init.d/couchbase-server start [2018-10-18 01:13:09,927] - [remote_util] [140543275300608] - INFO - command executed successfully [2018-10-18 01:13:09,927] - [remote_util] [140543275300608] - INFO - Starting couchbase-server [2018-10-18 01:13:09,927] - [remote_util] [140543275300608] - INFO - [ OK ] [2018-10-18 01:13:09,976] - [install] [140543275300608] - INFO - wait 5 seconds for Couchbase server to start [2018-10-18 01:13:13,504] - [remote_util] [140543266907904] - ERROR - Directory at /tmp DOES exist. Fx returns True [2018-10-18 01:13:13,607] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server [2018-10-18 01:13:13,657] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:13,791] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.106.209: /etc/init.d/couchbase-server start [2018-10-18 01:13:14,475] - [remote_util] [140543811811072] - ERROR - Directory at /tmp DOES exist. Fx returns True [2018-10-18 01:13:14,740] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server [2018-10-18 01:13:14,940] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:15,133] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.202: /etc/init.d/couchbase-server start [2018-10-18 01:13:15,243] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:13:15,245] - [remote_util] [140543266907904] - INFO - Starting couchbase-server [2018-10-18 01:13:15,245] - [remote_util] [140543266907904] - INFO - [ OK ] [2018-10-18 01:13:15,280] - [install] [140543266907904] - INFO - wait 5 seconds for Couchbase server to start [2018-10-18 01:13:16,867] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:16,867] - [remote_util] [140543811811072] - INFO - Starting couchbase-server [2018-10-18 01:13:16,867] - [remote_util] [140543811811072] - INFO - [ OK ] [2018-10-18 01:13:16,908] - [install] [140543811811072] - INFO - wait 5 seconds for Couchbase server to start [2018-10-18 01:13:19,540] - [remote_util] [140543769847552] - ERROR - Directory at /tmp DOES exist. Fx returns True [2018-10-18 01:13:19,639] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server [2018-10-18 01:13:19,719] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:19,824] - [remote_util] [140543769847552] - INFO - Running systemd command on this server [2018-10-18 01:13:19,824] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.208: systemctl start couchbase-server.service [2018-10-18 01:13:24,047] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:13:24,058] - [install] [140543769847552] - INFO - wait 5 seconds for Couchbase server to start [2018-10-18 01:13:29,070] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:13:29,073] - [remote_util] [140543224944384] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:13:29,074] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:13:29,075] - [remote_util] [140543250122496] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:13:29,304] - [remote_util] [140543250122496] - INFO - Connected to 172.23.106.204 [2018-10-18 01:13:29,319] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.208 [2018-10-18 01:13:29,320] - [remote_util] [140543224944384] - INFO - Connected to 172.23.106.209 [2018-10-18 01:13:29,398] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.202 [2018-10-18 01:13:29,591] - [remote_util] [140543811811072] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:13:29,890] - [remote_util] [140543250122496] - INFO - 172.23.106.204 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:13:29,917] - [remote_util] [140543224944384] - INFO - 172.23.106.209 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:13:30,054] - [remote_util] [140543811811072] - INFO - 172.23.106.208 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:13:30,183] - [remote_util] [140543769847552] - INFO - 172.23.106.202 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:13:33,906] - [install] [140543250122496] - INFO - quota for kv: 2068 MB [2018-10-18 01:13:33,906] - [rest_client] [140543250122496] - INFO - pools/default params : memoryQuota=2068 [2018-10-18 01:13:33,909] - [rest_client] [140543250122496] - INFO - /node/controller/setupServices params on 172.23.106.204: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator [2018-10-18 01:13:33,911] - [rest_client] [140543250122496] - INFO - settings/web params on 172.23.106.204:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:13:33,925] - [rest_client] [140543250122496] - INFO - Node version in cluster 4.6.3-4136-enterprise [2018-10-18 01:13:33,925] - [remote_util] [140543250122496] - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** [2018-10-18 01:13:33,934] - [install] [140543224944384] - INFO - quota for kv: 2068 MB [2018-10-18 01:13:33,934] - [rest_client] [140543224944384] - INFO - pools/default params : memoryQuota=2068 [2018-10-18 01:13:33,937] - [rest_client] [140543224944384] - INFO - /node/controller/setupServices params on 172.23.106.209: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator [2018-10-18 01:13:33,939] - [rest_client] [140543224944384] - INFO - settings/web params on 172.23.106.209:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:13:33,958] - [rest_client] [140543224944384] - INFO - Node version in cluster 4.6.3-4136-enterprise [2018-10-18 01:13:33,958] - [remote_util] [140543224944384] - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** [2018-10-18 01:13:34,068] - [install] [140543811811072] - INFO - quota for kv: 2013 MB [2018-10-18 01:13:34,069] - [rest_client] [140543811811072] - INFO - pools/default params : memoryQuota=2013 [2018-10-18 01:13:34,071] - [rest_client] [140543811811072] - INFO - /node/controller/setupServices params on 172.23.106.208: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator [2018-10-18 01:13:34,073] - [rest_client] [140543811811072] - INFO - settings/web params on 172.23.106.208:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:13:34,087] - [rest_client] [140543811811072] - INFO - Node version in cluster 4.6.3-4136-enterprise [2018-10-18 01:13:34,087] - [remote_util] [140543811811072] - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** [2018-10-18 01:13:34,208] - [install] [140543769847552] - INFO - quota for kv: 2068 MB [2018-10-18 01:13:34,208] - [rest_client] [140543769847552] - INFO - pools/default params : memoryQuota=2068 [2018-10-18 01:13:34,210] - [rest_client] [140543769847552] - INFO - /node/controller/setupServices params on 172.23.106.202: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator [2018-10-18 01:13:34,214] - [rest_client] [140543769847552] - INFO - settings/web params on 172.23.106.202:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:13:34,231] - [rest_client] [140543769847552] - INFO - Node version in cluster 4.6.3-4136-enterprise [2018-10-18 01:13:34,231] - [remote_util] [140543769847552] - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** [2018-10-18 01:13:34,264] - [remote_util] [140543250122496] - INFO - check if man installed on vm? [2018-10-18 01:13:34,264] - [remote_util] [140543250122496] - INFO - man page does not install man page on vm 172.23.106.204 Let do install it now [2018-10-18 01:13:34,264] - [remote_util] [140543250122496] - INFO - running command.raw on 172.23.106.204: yum install -y man [2018-10-18 01:13:34,303] - [remote_util] [140543224944384] - INFO - check if man installed on vm? [2018-10-18 01:13:34,303] - [remote_util] [140543224944384] - INFO - man page does not install man page on vm 172.23.106.209 Let do install it now [2018-10-18 01:13:34,303] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.106.209: yum install -y man [2018-10-18 01:13:34,400] - [remote_util] [140543811811072] - INFO - check if man installed on vm? [2018-10-18 01:13:34,400] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.208: rpm -qa | grep 'man-db' [2018-10-18 01:13:34,559] - [remote_util] [140543769847552] - INFO - check if man installed on vm? [2018-10-18 01:13:34,560] - [remote_util] [140543769847552] - INFO - man page does not install man page on vm 172.23.106.202 Let do install it now [2018-10-18 01:13:34,560] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.202: yum install -y man [2018-10-18 01:13:35,030] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:35,030] - [remote_util] [140543811811072] - INFO - ---> Run command unzip to check if it is ready on server 172.23.106.208 [2018-10-18 01:13:35,030] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.208: unzip [2018-10-18 01:13:35,078] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:35,079] - [remote_util] [140543811811072] - INFO - unzip command is ready [2018-10-18 01:13:35,079] - [remote_util] [140543811811072] - INFO - Check if ntp is installed [2018-10-18 01:13:35,079] - [remote_util] [140543811811072] - INFO - This OS version centos 7 [2018-10-18 01:13:35,079] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.208: systemctl status ntpd [2018-10-18 01:13:35,176] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:35,176] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.106.208: date [2018-10-18 01:13:35,216] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:13:35,370] - [remote_util] [140543811811072] - INFO - ['Thu Oct 18 01:13:35 PDT 2018'] IP: 172.23.106.208 [2018-10-18 01:13:35,384] - [remote_util] [140543250122496] - INFO - command executed successfully [2018-10-18 01:13:35,384] - [remote_util] [140543250122496] - INFO - ---> Run command unzip to check if it is ready on server 172.23.106.204 [2018-10-18 01:13:35,384] - [remote_util] [140543250122496] - INFO - running command.raw on 172.23.106.204: unzip [2018-10-18 01:13:35,466] - [remote_util] [140543250122496] - INFO - command executed successfully [2018-10-18 01:13:35,466] - [remote_util] [140543250122496] - INFO - unzip command is ready [2018-10-18 01:13:35,466] - [remote_util] [140543250122496] - INFO - Check if ntp is installed [2018-10-18 01:13:35,466] - [remote_util] [140543250122496] - INFO - This OS version centos release 6.5 (final) [2018-10-18 01:13:35,466] - [remote_util] [140543250122496] - INFO - running command.raw on 172.23.106.204: /etc/init.d/ntpd status [2018-10-18 01:13:35,564] - [remote_util] [140543250122496] - INFO - command executed successfully [2018-10-18 01:13:35,564] - [remote_util] [140543250122496] - INFO - running command.raw on 172.23.106.204: date [2018-10-18 01:13:35,613] - [remote_util] [140543250122496] - INFO - command executed successfully [2018-10-18 01:13:35,938] - [remote_util] [140543250122496] - INFO - ['Thu Oct 18 01:13:35 PDT 2018'] IP: 172.23.106.204 [2018-10-18 01:14:05,442] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:05,443] - [remote_util] [140543224944384] - INFO - ---> Run command unzip to check if it is ready on server 172.23.106.209 [2018-10-18 01:14:05,443] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.106.209: unzip [2018-10-18 01:14:05,492] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:05,492] - [remote_util] [140543224944384] - INFO - unzip command is ready [2018-10-18 01:14:05,493] - [remote_util] [140543224944384] - INFO - Check if ntp is installed [2018-10-18 01:14:05,493] - [remote_util] [140543224944384] - INFO - This OS version centos release 6.4 (final) [2018-10-18 01:14:05,493] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.106.209: /etc/init.d/ntpd status [2018-10-18 01:14:05,590] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:05,590] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.106.209: date [2018-10-18 01:14:05,639] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:05,965] - [remote_util] [140543224944384] - INFO - ['Thu Oct 18 01:14:05 PDT 2018'] IP: 172.23.106.209 [2018-10-18 01:14:06,801] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:06,801] - [remote_util] [140543769847552] - INFO - ---> Run command unzip to check if it is ready on server 172.23.106.202 [2018-10-18 01:14:06,801] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.202: unzip [2018-10-18 01:14:06,931] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:06,931] - [remote_util] [140543769847552] - INFO - unzip command is ready [2018-10-18 01:14:06,931] - [remote_util] [140543769847552] - INFO - Check if ntp is installed [2018-10-18 01:14:06,931] - [remote_util] [140543769847552] - INFO - This OS version centos release 6.8 (final) [2018-10-18 01:14:06,931] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.202: /etc/init.d/ntpd status [2018-10-18 01:14:07,063] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:07,063] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.202: date [2018-10-18 01:14:07,098] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:07,392] - [remote_util] [140543769847552] - INFO - ['Thu Oct 18 01:14:07 PDT 2018'] IP: 172.23.106.202 [2018-10-18 01:14:07,508] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:14:07,710] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:14:08,534] - [basetestcase] [140554369615616] - INFO - Atleast one of the nodes in the cluster is pre 5.0 version. Hence not creating rbac user for the cluster. RBAC is a 5.0 feature. [2018-10-18 01:14:08,535] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:14:08,737] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:14:09,545] - [remote_util] [140554369615616] - INFO - 172.23.106.202 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:14:09,545] - [basetestcase] [140554369615616] - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... [2018-10-18 01:14:14,675] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:14:14,808] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:14:15,380] - [remote_util] [140554369615616] - INFO - 172.23.106.204 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:14:15,380] - [basetestcase] [140554369615616] - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... [2018-10-18 01:14:20,507] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:14:20,605] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:14:20,849] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:14:21,310] - [remote_util] [140554369615616] - INFO - 172.23.106.208 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:14:21,310] - [basetestcase] [140554369615616] - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... [2018-10-18 01:14:26,321] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:14:26,487] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:14:27,085] - [remote_util] [140554369615616] - INFO - 172.23.106.209 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:14:27,085] - [basetestcase] [140554369615616] - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... [2018-10-18 01:14:32,220] - [newupgradebasetest] [140554369615616] - INFO - will install 6.5.0-1441 on ['172.23.106.63', '172.23.98.14'] [2018-10-18 01:14:32,224] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:14:32,225] - [remote_util] [140543266907904] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:14:32,341] - [remote_util] [140543266907904] - INFO - Connected to 172.23.98.14 [2018-10-18 01:14:32,392] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.63 [2018-10-18 01:14:32,613] - [remote_util] [140543266907904] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:14:32,897] - [remote_util] [140543769847552] - INFO - 172.23.106.63 *****In couchbase uninstall**** [2018-10-18 01:14:32,897] - [remote_util] [140543769847552] - INFO - CentOS [2018-10-18 01:14:33,012] - [remote_util] [140543266907904] - INFO - 172.23.98.14 *****In couchbase uninstall**** [2018-10-18 01:14:33,012] - [remote_util] [140543266907904] - INFO - CentOS [2018-10-18 01:14:33,021] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: killall -9 rpm [2018-10-18 01:14:33,057] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:33,057] - [remote_util] [140543769847552] - ERROR - rpm: no process killed [2018-10-18 01:14:33,057] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: rm -f /var/lib/rpm/.rpm.lock [2018-10-18 01:14:33,116] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: killall -9 rpm [2018-10-18 01:14:33,139] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:33,150] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:33,150] - [remote_util] [140543266907904] - ERROR - rpm: no process found [2018-10-18 01:14:33,150] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: rm -f /var/lib/rpm/.rpm.lock [2018-10-18 01:14:33,231] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:33,268] - [remote_util] [140543769847552] - INFO - running rpm -e to remove couchbase-server [2018-10-18 01:14:33,268] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: rpm -e couchbase-server [2018-10-18 01:14:33,336] - [remote_util] [140543266907904] - INFO - running rpm -e to remove couchbase-server [2018-10-18 01:14:33,336] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: rpm -e couchbase-server [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - ERROR - **** Uninstalling on server: 172.23.98.14 **** [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - ERROR - warning: file /opt/couchbase/var/lib/couchbase/ip_start: remove failed: No such file or directory [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave [2018-10-18 01:14:36,458] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: pkill -u couchbase [2018-10-18 01:14:36,477] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:36,477] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: ps -ef | grep couchbase [2018-10-18 01:14:36,574] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:36,574] - [remote_util] [140543266907904] - INFO - root 9029 8833 0 01:14 ? 00:00:00 bash -c ps -ef | grep couchbase [2018-10-18 01:14:36,575] - [remote_util] [140543266907904] - INFO - root 9035 9029 0 01:14 ? 00:00:00 grep couchbase [2018-10-18 01:14:37,920] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - ERROR - **** Uninstalling on server: 172.23.106.63 **** [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - ERROR - warning: /opt/couchbase/var/lib/couchbase/ip saved as /opt/couchbase/var/lib/couchbase/ip.rpmsave [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - ERROR - warning: /opt/couchbase/var/lib/couchbase/config/config.dat saved as /opt/couchbase/var/lib/couchbase/config/config.dat.rpmsave [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - ERROR - warning: /opt/couchbase/etc/couchdb/local.ini saved as /opt/couchbase/etc/couchdb/local.ini.rpmsave [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - INFO - **** Uninstalling on server: 172.23.106.63 **** [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - INFO - Stopping couchbase-server [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - INFO - [ OK ] [2018-10-18 01:14:37,921] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: pkill -u couchbase [2018-10-18 01:14:37,957] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:37,957] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: ps -ef | grep couchbase [2018-10-18 01:14:38,023] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:38,024] - [remote_util] [140543769847552] - INFO - root 22917 22723 0 01:14 ? 00:00:00 bash -c ps -ef | grep couchbase [2018-10-18 01:14:38,024] - [remote_util] [140543769847552] - INFO - root 22921 22917 0 01:14 ? 00:00:00 grep couchbase [2018-10-18 01:14:38,389] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: kill -9 $(ps aux | pgrep 'memcached') [2018-10-18 01:14:38,470] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:38,470] - [remote_util] [140543266907904] - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] [2018-10-18 01:14:38,470] - [remote_util] [140543266907904] - INFO - running command.raw on 172.23.98.14: ipcrm [2018-10-18 01:14:38,542] - [remote_util] [140543266907904] - INFO - command executed successfully [2018-10-18 01:14:40,099] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: kill -9 $(ps aux | pgrep 'memcached') [2018-10-18 01:14:40,197] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:40,197] - [remote_util] [140543769847552] - ERROR - kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec] [2018-10-18 01:14:40,198] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: ipcrm [2018-10-18 01:14:40,280] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:40,395] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:14:40,561] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:14:41,290] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:14:41,386] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:14:41,632] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:14:42,189] - [install] [140543769847552] - INFO - ********CouchbaseServerInstaller:install [2018-10-18 01:14:42,190] - [install] [140543224944384] - INFO - ********CouchbaseServerInstaller:install [2018-10-18 01:14:42,194] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:14:42,194] - [remote_util] [140543224944384] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:14:42,299] - [remote_util] [140543224944384] - INFO - Connected to 172.23.98.14 [2018-10-18 01:14:42,360] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.63 [2018-10-18 01:14:42,582] - [remote_util] [140543224944384] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:14:42,966] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:14:43,051] - [remote_util] [140543224944384] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:14:43,131] - [remote_util] [140543224944384] - INFO - Connected to 172.23.98.14 [2018-10-18 01:14:43,132] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.63 [2018-10-18 01:14:43,399] - [remote_util] [140543224944384] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:14:43,685] - [remote_util] [140543769847552] - INFO - Check if url http://172.23.120.24/builds/latestbuilds/couchbase-server/mad-hatter/1441/couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm is ok [2018-10-18 01:14:43,688] - [remote_util] [140543769847552] - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/mad-hatter/1441/couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm is live [2018-10-18 01:14:43,710] - [remote_util] [140543224944384] - INFO - Check if url http://172.23.120.24/builds/latestbuilds/couchbase-server/mad-hatter/1441/couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm is ok [2018-10-18 01:14:43,713] - [remote_util] [140543224944384] - INFO - This url http://172.23.120.24/builds/latestbuilds/couchbase-server/mad-hatter/1441/couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm is live [2018-10-18 01:14:43,721] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:14:43,808] - [remote_util] [140543224944384] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:14:43,886] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.63 [2018-10-18 01:14:43,888] - [remote_util] [140543224944384] - INFO - Connected to 172.23.98.14 [2018-10-18 01:14:44,130] - [remote_util] [140543224944384] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:14:44,410] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: /sbin/iptables -F [2018-10-18 01:14:44,492] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:44,492] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: /sbin/iptables -t nat -F [2018-10-18 01:14:44,532] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: /sbin/iptables -F [2018-10-18 01:14:44,574] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:44,574] - [remote_util] [140543769847552] - INFO - Connect to node: 172.23.106.63 as user: root [2018-10-18 01:14:44,665] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:44,665] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: /sbin/iptables -t nat -F [2018-10-18 01:14:44,740] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.63 as root [2018-10-18 01:14:44,740] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D [2018-10-18 01:14:44,746] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:44,746] - [remote_util] [140543224944384] - INFO - Connect to node: 172.23.98.14 as user: root [2018-10-18 01:14:44,775] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:44,775] - [remote_util] [140543769847552] - ERROR - mv: cannot stat `core.*': No such file or directory [2018-10-18 01:14:44,776] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `backup_crash': Is a directory [2018-10-18 01:14:44,776] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `cb_Xz1h': Is a directory [2018-10-18 01:14:44,776] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `d1': Is a directory [2018-10-18 01:14:44,776] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `d2': Is a directory [2018-10-18 01:14:44,776] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `d3': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `d4': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `d5': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `d6': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `entbackup': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `HOTFIX-CBSE-1341': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `measure-sched-delays': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmp17jDp7': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmp1rfaT9': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmp7Xoepq': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmp84DA25': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpf50pDr': Is a directory [2018-10-18 01:14:44,777] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmplehLY8': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpOSOIe_': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpVtPBYn': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpxgFJnH': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpXgy2Fq': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpy8lPsG': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - ERROR - rm: cannot remove `tmpZEVcGA': Is a directory [2018-10-18 01:14:44,778] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: cd /tmp;wget -q -O couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/mad-hatter/1441/couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm;cd /tmp;ls -lh [2018-10-18 01:14:44,842] - [remote_util] [140543224944384] - INFO - Connected to 172.23.98.14 as root [2018-10-18 01:14:44,843] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: cd /tmp ; D=$(mktemp -d cb_XXXX) ; mv couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm $D ; mv core.* $D ; rm -f * ; mv $D/* . ; rmdir $D [2018-10-18 01:14:45,041] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - mv: cannot stat ‘core.*’: No such file or directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘_12#-test’: Is a directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘cb_CTyw’: Is a directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘entbackup’: Is a directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘hsperfdata_couchbase’: Is a directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘l1234’: Is a directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘logs’: Is a directory [2018-10-18 01:14:45,042] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘LOGS’: Is a directory [2018-10-18 01:14:45,043] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘non-exist-dir’: Is a directory [2018-10-18 01:14:45,043] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘spa ce’: Is a directory [2018-10-18 01:14:45,043] - [remote_util] [140543224944384] - ERROR - rm: cannot remove ‘systemd-private-179338c05b494152b40f023d0d11d216-ntpd.service-Mg8H8Q’: Is a directory [2018-10-18 01:14:45,043] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: cd /tmp;wget -q -O couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm http://172.23.120.24/builds/latestbuilds/couchbase-server/mad-hatter/1441/couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm;cd /tmp;ls -lh [2018-10-18 01:14:52,752] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:52,752] - [remote_util] [140543224944384] - INFO - total 245M [2018-10-18 01:14:52,752] - [remote_util] [140543224944384] - INFO - drwxr-xr-x. 4 root root 4.0K Oct 11 23:52 _12#-test [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - -rw-r--r--. 1 root root 245M Oct 16 10:40 couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxrwx---. 4 root root 44 Oct 17 15:30 entbackup [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxr-x---. 2 couchbase couchbase 6 Oct 17 00:48 hsperfdata_couchbase [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxr-xr-x. 4 root root 4.0K Oct 11 23:53 l1234 [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxr-xr-x. 4 root root 4.0K Oct 11 23:59 logs [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxr-xr-x. 4 root root 4.0K Oct 11 23:55 LOGS [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxr-xr-x. 4 root root 4.0K Oct 11 23:50 non-exist-dir [2018-10-18 01:14:52,753] - [remote_util] [140543224944384] - INFO - drwxr-xr-x. 4 root root 4.0K Oct 11 23:56 spa ce [2018-10-18 01:14:52,754] - [remote_util] [140543224944384] - INFO - drwx------. 3 root root 16 Oct 4 08:48 systemd-private-179338c05b494152b40f023d0d11d216-ntpd.service-Mg8H8Q [2018-10-18 01:14:52,771] - [remote_util] [140543224944384] - INFO - *****install server *** [2018-10-18 01:14:52,771] - [remote_util] [140543224944384] - INFO - deliverable_type : rpm [2018-10-18 01:14:52,771] - [remote_util] [140543224944384] - INFO - /tmp/couchbase-server-enterprise-6.5.0-1441-centos7.x86_64.rpm or /tmp/ [2018-10-18 01:14:52,771] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: /sbin/sysctl vm.swappiness=0 [2018-10-18 01:14:52,789] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:52,789] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: cat /etc/redhat-release [2018-10-18 01:14:52,870] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:14:52,870] - [remote_util] [140543224944384] - INFO - CentOS Linux release 7.3.1611 (Core) [2018-10-18 01:14:52,870] - [remote_util] [140543224944384] - INFO - no need to install pkgconfig [2018-10-18 01:14:54,575] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:54,575] - [remote_util] [140543769847552] - INFO - total 233M [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxr-xr-x 21 root root 4.0K Oct 13 14:38 backup_crash [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - -rw-r--r-- 1 root root 233M Oct 16 10:34 couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxrwx--- 2 couchbase couchbase 4.0K Jun 15 03:45 d1 [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxrwx--- 2 couchbase couchbase 4.0K Jun 15 03:45 d2 [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxrwx--- 2 couchbase couchbase 4.0K Jun 15 03:45 d3 [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxrwx--- 2 couchbase couchbase 4.0K Jun 15 03:45 d4 [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxrwx--- 2 couchbase couchbase 4.0K Jun 15 03:45 d5 [2018-10-18 01:14:54,576] - [remote_util] [140543769847552] - INFO - drwxrwx--- 2 couchbase couchbase 4.0K Jun 15 03:45 d6 [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwxrwx--- 4 root root 4.0K Oct 17 18:33 entbackup [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwxr-xr-x 2 root root 4.0K Aug 17 2014 HOTFIX-CBSE-1341 [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwxrwxr-x 3 1000 1000 4.0K Jan 29 2017 measure-sched-delays [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K Mar 26 2015 tmp17jDp7 [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K Jun 5 04:29 tmp1rfaT9 [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmp7Xoepq [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K Mar 26 2015 tmp84DA25 [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K Aug 31 2016 tmpf50pDr [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmplehLY8 [2018-10-18 01:14:54,577] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmpOSOIe_ [2018-10-18 01:14:54,578] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmpVtPBYn [2018-10-18 01:14:54,578] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K Apr 8 2016 tmpxgFJnH [2018-10-18 01:14:54,578] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmpXgy2Fq [2018-10-18 01:14:54,578] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K May 2 2016 tmpy8lPsG [2018-10-18 01:14:54,578] - [remote_util] [140543769847552] - INFO - drwx------ 2 root root 4.0K Apr 7 2016 tmpZEVcGA [2018-10-18 01:14:54,621] - [remote_util] [140543769847552] - INFO - *****install server *** [2018-10-18 01:14:54,621] - [remote_util] [140543769847552] - INFO - deliverable_type : rpm [2018-10-18 01:14:54,621] - [remote_util] [140543769847552] - INFO - /tmp/couchbase-server-enterprise-6.5.0-1441-centos6.x86_64.rpm or /tmp/ [2018-10-18 01:14:54,622] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: /sbin/sysctl vm.swappiness=0 [2018-10-18 01:14:54,640] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:54,640] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: cat /etc/redhat-release [2018-10-18 01:14:54,722] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:54,722] - [remote_util] [140543769847552] - INFO - CentOS release 6.4 (Final) [2018-10-18 01:14:54,723] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: rpm -qa | grep pkgconfig [2018-10-18 01:14:55,080] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:14:55,080] - [remote_util] [140543769847552] - INFO - pkgconfig-0.23-9.1.el6.x86_64 [2018-10-18 01:15:19,351] - [remote_util] [140543224944384] - ERROR - Directory at /tmp DOES exist. Fx returns True [2018-10-18 01:15:19,452] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server [2018-10-18 01:15:19,532] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:15:19,635] - [remote_util] [140543224944384] - INFO - Running systemd command on this server [2018-10-18 01:15:19,635] - [remote_util] [140543224944384] - INFO - running command.raw on 172.23.98.14: systemctl start couchbase-server.service [2018-10-18 01:15:19,701] - [remote_util] [140543224944384] - INFO - command executed successfully [2018-10-18 01:15:19,734] - [install] [140543224944384] - INFO - wait 5 seconds for Couchbase server to start [2018-10-18 01:15:30,751] - [remote_util] [140543769847552] - ERROR - Directory at /tmp DOES exist. Fx returns True [2018-10-18 01:15:30,838] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: sed -i 's/export PATH/export PATH\nexport CBFT_ENV_OPTIONS=bleveMaxResultWindow=10000000/' /opt/couchbase/bin/couchbase-server [2018-10-18 01:15:30,919] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:15:31,056] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: /etc/init.d/couchbase-server start [2018-10-18 01:15:33,709] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:15:33,709] - [remote_util] [140543769847552] - INFO - Starting couchbase-server [2018-10-18 01:15:33,709] - [remote_util] [140543769847552] - INFO - [ OK ] [2018-10-18 01:15:33,745] - [install] [140543769847552] - INFO - wait 5 seconds for Couchbase server to start [2018-10-18 01:15:38,863] - [remote_util] [140543811811072] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:15:38,864] - [remote_util] [140543769847552] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:15:39,030] - [remote_util] [140543769847552] - INFO - Connected to 172.23.106.63 [2018-10-18 01:15:39,031] - [remote_util] [140543811811072] - INFO - Connected to 172.23.98.14 [2018-10-18 01:15:39,283] - [remote_util] [140543811811072] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:15:39,658] - [remote_util] [140543769847552] - INFO - 172.23.106.63 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:15:39,659] - [rest_client] [140543769847552] - ERROR - socket error while connecting to http://172.23.106.63:8091/nodes/self error [Errno 111] Connection refused [2018-10-18 01:15:39,740] - [remote_util] [140543811811072] - INFO - 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:15:43,761] - [install] [140543811811072] - INFO - quota for kv: 2013 MB [2018-10-18 01:15:43,761] - [rest_client] [140543811811072] - INFO - pools/default params : memoryQuota=2013 [2018-10-18 01:15:43,768] - [rest_client] [140543811811072] - INFO - /node/controller/setupServices params on 172.23.98.14: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator [2018-10-18 01:15:43,775] - [rest_client] [140543811811072] - INFO - settings/web params on 172.23.98.14:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:15:43,838] - [rest_client] [140543811811072] - INFO - Node version in cluster 6.5.0-1441-enterprise [2018-10-18 01:15:43,838] - [remote_util] [140543811811072] - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** [2018-10-18 01:15:44,129] - [remote_util] [140543811811072] - INFO - check if man installed on vm? [2018-10-18 01:15:44,129] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.98.14: rpm -qa | grep 'man-db' [2018-10-18 01:15:44,809] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:15:44,809] - [remote_util] [140543811811072] - INFO - ---> Run command unzip to check if it is ready on server 172.23.98.14 [2018-10-18 01:15:44,810] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.98.14: unzip [2018-10-18 01:15:44,828] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:15:44,828] - [remote_util] [140543811811072] - INFO - unzip command is ready [2018-10-18 01:15:44,828] - [remote_util] [140543811811072] - INFO - Check if ntp is installed [2018-10-18 01:15:44,828] - [remote_util] [140543811811072] - INFO - This OS version centos 7 [2018-10-18 01:15:44,828] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.98.14: systemctl status ntpd [2018-10-18 01:15:44,925] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:15:44,925] - [remote_util] [140543811811072] - INFO - running command.raw on 172.23.98.14: date [2018-10-18 01:15:44,967] - [remote_util] [140543811811072] - INFO - command executed successfully [2018-10-18 01:15:45,145] - [remote_util] [140543811811072] - INFO - ['Thu Oct 18 01:15:45 PDT 2018'] IP: 172.23.98.14 [2018-10-18 01:15:46,689] - [install] [140543769847552] - INFO - quota for kv: 2068 MB [2018-10-18 01:15:46,689] - [rest_client] [140543769847552] - INFO - pools/default params : memoryQuota=2068 [2018-10-18 01:15:46,699] - [rest_client] [140543769847552] - INFO - /node/controller/setupServices params on 172.23.106.63: 8091:services=kv&password=password&hostname=127.0.0.1%3A8091&user=Administrator [2018-10-18 01:15:46,707] - [rest_client] [140543769847552] - INFO - settings/web params on 172.23.106.63:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:15:46,807] - [rest_client] [140543769847552] - INFO - Node version in cluster 6.5.0-1441-enterprise [2018-10-18 01:15:46,807] - [remote_util] [140543769847552] - INFO - ***** set NS_SERVER_CBAUTH env in linux ***** [2018-10-18 01:15:47,135] - [remote_util] [140543769847552] - INFO - check if man installed on vm? [2018-10-18 01:15:47,135] - [remote_util] [140543769847552] - INFO - man page does not install man page on vm 172.23.106.63 Let do install it now [2018-10-18 01:15:47,136] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: yum install -y man [2018-10-18 01:16:18,271] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:16:18,271] - [remote_util] [140543769847552] - INFO - ---> Run command unzip to check if it is ready on server 172.23.106.63 [2018-10-18 01:16:18,272] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: unzip [2018-10-18 01:16:18,321] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:16:18,321] - [remote_util] [140543769847552] - INFO - unzip command is ready [2018-10-18 01:16:18,321] - [remote_util] [140543769847552] - INFO - Check if ntp is installed [2018-10-18 01:16:18,321] - [remote_util] [140543769847552] - INFO - This OS version centos release 6.4 (final) [2018-10-18 01:16:18,322] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: /etc/init.d/ntpd status [2018-10-18 01:16:18,419] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:16:18,420] - [remote_util] [140543769847552] - INFO - running command.raw on 172.23.106.63: date [2018-10-18 01:16:18,501] - [remote_util] [140543769847552] - INFO - command executed successfully [2018-10-18 01:16:18,831] - [remote_util] [140543769847552] - INFO - ['Thu Oct 18 01:16:18 PDT 2018'] IP: 172.23.106.63 [2018-10-18 01:16:18,947] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:16:19,112] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:16:19,679] - [basetestcase] [140554369615616] - INFO - Atleast one of the nodes in the cluster is pre 5.0 version. Hence not creating rbac user for the cluster. RBAC is a 5.0 feature. [2018-10-18 01:16:19,680] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:16:19,845] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:16:20,455] - [remote_util] [140554369615616] - INFO - 172.23.106.63 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:16:20,456] - [basetestcase] [140554369615616] - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... [2018-10-18 01:16:25,587] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:16:25,684] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:16:25,929] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:26,366] - [remote_util] [140554369615616] - INFO - 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:16:26,366] - [basetestcase] [140554369615616] - INFO - sleep for 5 secs. sleep 5 seconds to let cb up completely ... [2018-10-18 01:16:31,501] - [rest_client] [140554369615616] - INFO - /diag/eval status on 172.23.106.202:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-10-18 01:16:31,502] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:16:31,704] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:16:32,444] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:32,510] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:32,511] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:32,515] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:16:32,717] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:16:33,449] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:33,547] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:33,548] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:33,663] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:16:33,796] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:16:34,289] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:34,386] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:34,386] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:34,501] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:16:34,634] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:16:35,130] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:35,227] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:35,227] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:35,345] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:16:35,442] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:16:35,686] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:36,081] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:36,162] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:36,162] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:36,278] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:16:36,375] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:16:36,614] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:36,969] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:37,049] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:37,050] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:37,165] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:16:37,331] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:16:37,853] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:37,951] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:37,951] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:38,066] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:16:38,232] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:16:38,745] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:38,842] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:38,842] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:38,961] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:16:39,127] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:16:39,676] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:39,741] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:39,741] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:39,857] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:16:40,022] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:16:40,558] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:40,656] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:40,656] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:40,771] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:16:40,868] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:16:41,113] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:41,489] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: zgrep "panic:" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:41,570] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:41,570] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:41,685] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:16:41,766] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:16:42,011] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:42,373] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: zgrep "non-recoverable error from xmem client. response status=KEY_ENOENT" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:42,454] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:42,454] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:42,568] - [xdcrnewbasetests] [140554369615616] - INFO - {'172.23.106.63': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.98.14': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.106.208': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.106.209': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.106.204': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.106.202': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}} [2018-10-18 01:16:42,579] - [rest_client] [140554369615616] - INFO - /diag/eval status on 172.23.106.202:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2018-10-18 01:16:42,580] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:16:42,782] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:16:43,489] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:43,554] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:43,554] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:43,670] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:16:43,804] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:16:44,296] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:44,393] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:44,393] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:44,509] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:16:44,605] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:16:44,850] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:45,239] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:45,319] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:45,320] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:45,435] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:16:45,601] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:16:46,112] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:46,210] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:46,211] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:46,326] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:16:46,492] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:16:47,025] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:47,091] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:47,091] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:47,206] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:16:47,305] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:16:47,576] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:16:47,981] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: zgrep "Try to fix Pipeline" "/opt/couchbase/var/lib/couchbase/logs"/goxdcr.log* | wc -l [2018-10-18 01:16:48,062] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:16:48,062] - [xdcrnewbasetests] [140554369615616] - INFO - 0 [2018-10-18 01:16:48,176] - [xdcrnewbasetests] [140554369615616] - INFO - {'172.23.106.63': 0, '172.23.98.14': 0, '172.23.106.208': 0, '172.23.106.209': 0, '172.23.106.204': 0, '172.23.106.202': 0} [2018-10-18 01:16:48,176] - [xdcrnewbasetests] [140554369615616] - INFO - ==== XDCRNewbasetests setup is started for test #16 online_cluster_upgrade ==== [2018-10-18 01:16:48,177] - [xdcrnewbasetests] [140554369615616] - INFO - removing xdcr/nodes settings [2018-10-18 01:16:48,199] - [xdcrnewbasetests] [140554369615616] - INFO - cleanup [ip:172.23.106.202 port:8091 ssh_username:root, ip:172.23.106.204 port:8091 ssh_username:root] [2018-10-18 01:16:48,233] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.202:8091 [2018-10-18 01:16:48,236] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.202:8091 is running [2018-10-18 01:16:48,257] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.204:8091 [2018-10-18 01:16:48,260] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.204:8091 is running [2018-10-18 01:16:48,260] - [xdcrnewbasetests] [140554369615616] - INFO - removing xdcr/nodes settings [2018-10-18 01:16:48,275] - [xdcrnewbasetests] [140554369615616] - INFO - cleanup [ip:172.23.106.208 port:8091 ssh_username:root, ip:172.23.106.209 port:8091 ssh_username:root] [2018-10-18 01:16:48,295] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.208:8091 [2018-10-18 01:16:48,298] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.208:8091 is running [2018-10-18 01:16:48,348] - [cluster_helper] [140554369615616] - INFO - waiting for ns_server @ 172.23.106.209:8091 [2018-10-18 01:16:48,352] - [cluster_helper] [140554369615616] - INFO - ns_server @ 172.23.106.209:8091 is running [2018-10-18 01:16:48,352] - [xdcrnewbasetests] [140554369615616] - INFO - Initializing all clusters... [2018-10-18 01:16:48,356] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:16:48,558] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:16:49,323] - [remote_util] [140554369615616] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-10-18 01:16:49,323] - [xdcrnewbasetests] [140554369615616] - INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2018-10-18 01:16:49,325] - [rest_client] [140554369615616] - INFO - /diag/eval status on 172.23.106.202:8091: True content: [4,6] command: cluster_compat_mode:get_compat_version(). [2018-10-18 01:16:50,186] - [task] [140543224944384] - INFO - server: ip:172.23.106.202 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'210', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.106.202:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'4.6.3-4136-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 262150, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3005145088, 'memoryTotal': 4069212160, 'memoryQuota': 2068, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-10-18 01:16:50,186] - [rest_client] [140543224944384] - INFO - pools/default params : memoryQuota=2069 [2018-10-18 01:16:50,190] - [rest_client] [140543224944384] - INFO - settings/web params on 172.23.106.202:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:16:50,195] - [remote_util] [140543224944384] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:16:50,396] - [remote_util] [140543224944384] - INFO - Connected to 172.23.106.202 [2018-10-18 01:16:51,157] - [remote_util] [140543224944384] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-10-18 01:16:51,159] - [rest_client] [140543224944384] - INFO - /diag/eval status on 172.23.106.202:8091: True content: [4,6] command: cluster_compat_mode:get_compat_version(). [2018-10-18 01:16:51,159] - [rest_client] [140543224944384] - INFO - settings/indexes params : storageMode=forestdb [2018-10-18 01:16:51,180] - [task] [140543224944384] - INFO - server: ip:172.23.106.204 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'219', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3880, 'hostname': u'172.23.106.204:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'4.6.3-4136-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 262150, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3657109504, 'memoryTotal': 4069449728, 'memoryQuota': 2068, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-10-18 01:16:51,182] - [rest_client] [140543224944384] - INFO - pools/default params : memoryQuota=2069 [2018-10-18 01:16:51,188] - [rest_client] [140543224944384] - INFO - settings/web params on 172.23.106.204:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:16:51,191] - [remote_util] [140543224944384] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:16:51,324] - [remote_util] [140543224944384] - INFO - Connected to 172.23.106.204 [2018-10-18 01:16:51,894] - [remote_util] [140543224944384] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-10-18 01:16:51,896] - [rest_client] [140543224944384] - INFO - /diag/eval status on 172.23.106.204:8091: True content: [4,6] command: cluster_compat_mode:get_compat_version(). [2018-10-18 01:16:51,896] - [rest_client] [140543224944384] - INFO - settings/indexes params : storageMode=forestdb [2018-10-18 01:16:52,910] - [task] [140543224944384] - INFO - adding node 172.23.106.204:8091 to cluster [2018-10-18 01:16:52,910] - [rest_client] [140543224944384] - INFO - adding remote node @172.23.106.204:8091 to this cluster @172.23.106.202:8091 [2018-10-18 01:16:55,417] - [rest_client] [140543224944384] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.204,ns_1@172.23.106.202'} [2018-10-18 01:16:55,427] - [rest_client] [140543224944384] - INFO - rebalance operation started [2018-10-18 01:16:55,433] - [rest_client] [140543224944384] - INFO - rebalance percentage : 0.00 % [2018-10-18 01:16:55,434] - [task] [140543224944384] - INFO - Rebalance - status: running, progress: 0.00% [2018-10-18 01:17:05,453] - [task] [140543224944384] - INFO - Rebalance - status: none, progress: 100.00% [2018-10-18 01:17:05,457] - [task] [140543224944384] - INFO - rebalancing was completed with progress: 100% in 10.0291440487 sec [2018-10-18 01:17:05,462] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:17:05,558] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:17:05,795] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:17:06,160] - [remote_util] [140554369615616] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-10-18 01:17:06,160] - [xdcrnewbasetests] [140554369615616] - INFO - Running in compatibility mode, not enabled diag/eval for non-local hosts [2018-10-18 01:17:06,162] - [rest_client] [140554369615616] - INFO - /diag/eval status on 172.23.106.208:8091: True content: [4,6] command: cluster_compat_mode:get_compat_version(). [2018-10-18 01:17:06,204] - [task] [140543811811072] - INFO - server: ip:172.23.106.208 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'223', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.106.208:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'4.6.3-4136-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 262150, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3468029952, 'memoryTotal': 3961200640, 'memoryQuota': 2013, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-10-18 01:17:06,207] - [rest_client] [140543811811072] - INFO - pools/default params : memoryQuota=2014 [2018-10-18 01:17:06,209] - [rest_client] [140543811811072] - INFO - settings/web params on 172.23.106.208:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:17:06,213] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:17:06,330] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.208 [2018-10-18 01:17:06,528] - [remote_util] [140543811811072] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:17:06,914] - [remote_util] [140543811811072] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-10-18 01:17:06,916] - [rest_client] [140543811811072] - INFO - /diag/eval status on 172.23.106.208:8091: True content: [4,6] command: cluster_compat_mode:get_compat_version(). [2018-10-18 01:17:06,916] - [rest_client] [140543811811072] - INFO - settings/indexes params : storageMode=forestdb [2018-10-18 01:17:06,934] - [task] [140543811811072] - INFO - server: ip:172.23.106.209 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'227', 'mcdMemoryReserved': 3104, 'storageTotalRam': 3881, 'hostname': u'172.23.106.209:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'4.6.3-4136-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 262150, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3641905152, 'memoryTotal': 4069695488, 'memoryQuota': 2068, 'mcdMemoryAllocated': 3104, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2018-10-18 01:17:06,934] - [rest_client] [140543811811072] - INFO - pools/default params : memoryQuota=2069 [2018-10-18 01:17:06,939] - [rest_client] [140543811811072] - INFO - settings/web params on 172.23.106.209:8091:username=Administrator&password=password&port=8091 [2018-10-18 01:17:06,945] - [remote_util] [140543811811072] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:17:07,110] - [remote_util] [140543811811072] - INFO - Connected to 172.23.106.209 [2018-10-18 01:17:07,636] - [remote_util] [140543811811072] - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases [2018-10-18 01:17:07,639] - [rest_client] [140543811811072] - INFO - /diag/eval status on 172.23.106.209:8091: True content: [4,6] command: cluster_compat_mode:get_compat_version(). [2018-10-18 01:17:07,639] - [rest_client] [140543811811072] - INFO - settings/indexes params : storageMode=forestdb [2018-10-18 01:17:08,656] - [task] [140543811811072] - INFO - adding node 172.23.106.209:8091 to cluster [2018-10-18 01:17:08,656] - [rest_client] [140543811811072] - INFO - adding remote node @172.23.106.209:8091 to this cluster @172.23.106.208:8091 [2018-10-18 01:17:11,942] - [rest_client] [140543811811072] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.209,ns_1@172.23.106.208'} [2018-10-18 01:17:11,950] - [rest_client] [140543811811072] - INFO - rebalance operation started [2018-10-18 01:17:11,952] - [rest_client] [140543811811072] - INFO - rebalance percentage : 0.00 % [2018-10-18 01:17:11,952] - [task] [140543811811072] - INFO - Rebalance - status: running, progress: 0.00% [2018-10-18 01:17:21,970] - [task] [140543811811072] - INFO - Rebalance - status: none, progress: 100.00% [2018-10-18 01:17:21,975] - [task] [140543811811072] - INFO - rebalancing was completed with progress: 100% in 10.0244457722 sec [2018-10-18 01:17:21,993] - [internal_user] [140554369615616] - INFO - At least one of the node in the cluster is on pre-spock version. Not creating user since RBAC is a spock feature. [2018-10-18 01:17:22,006] - [internal_user] [140554369615616] - INFO - At least one of the node in the cluster is on pre-spock version. Not creating user since RBAC is a spock feature. [2018-10-18 01:17:22,013] - [xdcrnewbasetests] [140554369615616] - INFO - ==== XDCRNewbasetests setup is finished for test #16 online_cluster_upgrade ==== [2018-10-18 01:17:22,502] - [rest_client] [140543224944384] - INFO - http://172.23.106.202:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=689&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2018-10-18 01:17:22,508] - [rest_client] [140543224944384] - INFO - 0.01 seconds to create bucket default [2018-10-18 01:17:22,508] - [bucket_helper] [140543224944384] - INFO - waiting for memcached bucket : default in 172.23.106.202 to accept set ops [2018-10-18 01:17:24,150] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:17:24,212] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:24,223] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:17:24,278] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:24,331] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:17:24,385] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:24,420] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:17:24,468] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:24,494] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:24,631] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:24,804] - [task] [140543224944384] - INFO - bucket 'default' was created with per node RAM quota: 689 [2018-10-18 01:17:25,829] - [rest_client] [140543224944384] - INFO - http://172.23.106.202:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=689&proxyPort=11211&authType=sasl&name=sasl_bucket_1&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=password&replicaIndex=1 [2018-10-18 01:17:25,835] - [rest_client] [140543224944384] - INFO - 0.01 seconds to create bucket sasl_bucket_1 [2018-10-18 01:17:25,836] - [bucket_helper] [140543224944384] - INFO - waiting for memcached bucket : sasl_bucket_1 in 172.23.106.202 to accept set ops [2018-10-18 01:17:27,449] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:17:27,501] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:27,518] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:17:27,570] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:27,646] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:17:27,697] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:27,715] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:17:27,793] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:27,814] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:27,928] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:28,118] - [task] [140543224944384] - INFO - bucket 'sasl_bucket_1' was created with per node RAM quota: 689 [2018-10-18 01:17:29,139] - [rest_client] [140543224944384] - INFO - http://172.23.106.202:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=689&proxyPort=11217&authType=none&name=standard_bucket_1&flushEnabled=1&replicaNumber=1&compressionMode=passive&replicaIndex=1 [2018-10-18 01:17:29,148] - [rest_client] [140543224944384] - INFO - 0.01 seconds to create bucket standard_bucket_1 [2018-10-18 01:17:29,148] - [bucket_helper] [140543224944384] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.106.202 to accept set ops [2018-10-18 01:17:30,290] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.202:11210 standard_bucket_1 [2018-10-18 01:17:30,352] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:30,376] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.204:11210 standard_bucket_1 [2018-10-18 01:17:30,426] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:30,475] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.202:11210 standard_bucket_1 [2018-10-18 01:17:30,520] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:30,533] - [data_helper] [140543224944384] - INFO - creating direct client 172.23.106.204:11210 standard_bucket_1 [2018-10-18 01:17:30,584] - [data_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:30,605] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:30,770] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:31,046] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:31,201] - [bucket_helper] [140543224944384] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:31,356] - [task] [140543224944384] - INFO - bucket 'standard_bucket_1' was created with per node RAM quota: 689 [2018-10-18 01:17:31,999] - [rest_client] [140543811811072] - INFO - http://172.23.106.208:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=671&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2018-10-18 01:17:32,003] - [rest_client] [140543811811072] - INFO - 0.00 seconds to create bucket default [2018-10-18 01:17:32,003] - [bucket_helper] [140543811811072] - INFO - waiting for memcached bucket : default in 172.23.106.208 to accept set ops [2018-10-18 01:17:33,112] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.208:11210 default [2018-10-18 01:17:33,149] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,162] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.209:11210 default [2018-10-18 01:17:33,224] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,253] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.208:11210 default [2018-10-18 01:17:33,285] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,299] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.209:11210 default [2018-10-18 01:17:33,344] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,358] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,635] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,769] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:33,948] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:34,078] - [task] [140543811811072] - INFO - bucket 'default' was created with per node RAM quota: 671 [2018-10-18 01:17:35,096] - [rest_client] [140543811811072] - INFO - http://172.23.106.208:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=671&proxyPort=11211&authType=sasl&name=sasl_bucket_1&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=password&replicaIndex=1 [2018-10-18 01:17:35,101] - [rest_client] [140543811811072] - INFO - 0.01 seconds to create bucket sasl_bucket_1 [2018-10-18 01:17:35,102] - [bucket_helper] [140543811811072] - INFO - waiting for memcached bucket : sasl_bucket_1 in 172.23.106.208 to accept set ops [2018-10-18 01:17:36,679] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.208:11210 sasl_bucket_1 [2018-10-18 01:17:36,711] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:36,725] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.209:11210 sasl_bucket_1 [2018-10-18 01:17:36,789] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:36,818] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.208:11210 sasl_bucket_1 [2018-10-18 01:17:36,848] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:36,860] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.209:11210 sasl_bucket_1 [2018-10-18 01:17:36,922] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:36,938] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:37,887] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:38,005] - [task] [140543811811072] - INFO - bucket 'sasl_bucket_1' was created with per node RAM quota: 671 [2018-10-18 01:17:39,021] - [rest_client] [140543811811072] - INFO - http://172.23.106.208:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=671&proxyPort=11217&authType=none&name=standard_bucket_1&flushEnabled=1&replicaNumber=1&compressionMode=passive&replicaIndex=1 [2018-10-18 01:17:39,029] - [rest_client] [140543811811072] - INFO - 0.01 seconds to create bucket standard_bucket_1 [2018-10-18 01:17:39,030] - [bucket_helper] [140543811811072] - INFO - waiting for memcached bucket : standard_bucket_1 in 172.23.106.208 to accept set ops [2018-10-18 01:17:40,130] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.208:11210 standard_bucket_1 [2018-10-18 01:17:40,165] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:40,179] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.209:11210 standard_bucket_1 [2018-10-18 01:17:40,232] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:40,263] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.208:11210 standard_bucket_1 [2018-10-18 01:17:40,301] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:40,313] - [data_helper] [140543811811072] - INFO - creating direct client 172.23.106.209:11210 standard_bucket_1 [2018-10-18 01:17:40,365] - [data_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:40,382] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:40,615] - [bucket_helper] [140543811811072] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:17:40,751] - [task] [140543811811072] - INFO - bucket 'standard_bucket_1' was created with per node RAM quota: 671 [2018-10-18 01:17:40,757] - [rest_client] [140554369615616] - INFO - adding remote cluster hostname:172.23.106.208:8091 with username:password Administrator:password name:remote_cluster_C1-C2 to source node: 172.23.106.202:8091 [2018-10-18 01:17:40,781] - [rest_client] [140554369615616] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C1-C2 with settings {} [2018-10-18 01:17:40,818] - [rest_client] [140554369615616] - INFO - Replication created with id: 44374c76dbff7c3aecbf399aeef22116/default/default [2018-10-18 01:17:50,833] - [rest_client] [140554369615616] - INFO - starting continuous replication type:xmem from sasl_bucket_1 to sasl_bucket_1 in the remote cluster remote_cluster_C1-C2 with settings {} [2018-10-18 01:17:50,875] - [rest_client] [140554369615616] - INFO - Replication created with id: 44374c76dbff7c3aecbf399aeef22116/sasl_bucket_1/sasl_bucket_1 [2018-10-18 01:18:00,891] - [rest_client] [140554369615616] - INFO - adding remote cluster hostname:172.23.106.202:8091 with username:password Administrator:password name:remote_cluster_C2-C1 to source node: 172.23.106.208:8091 [2018-10-18 01:18:00,910] - [rest_client] [140554369615616] - INFO - starting continuous replication type:xmem from default to default in the remote cluster remote_cluster_C2-C1 with settings {} [2018-10-18 01:18:00,941] - [rest_client] [140554369615616] - INFO - Replication created with id: cf3210709dd594147da343c8ee264a60/default/default [2018-10-18 01:18:10,952] - [rest_client] [140554369615616] - INFO - starting continuous replication type:xmem from standard_bucket_1 to standard_bucket_1 in the remote cluster remote_cluster_C2-C1 with settings {} [2018-10-18 01:18:10,980] - [rest_client] [140554369615616] - INFO - Replication created with id: cf3210709dd594147da343c8ee264a60/standard_bucket_1/standard_bucket_1 [2018-10-18 01:18:20,995] - [rest_client] [140554369615616] - INFO - starting continuous replication type:xmem from sasl_bucket_1 to sasl_bucket_1 in the remote cluster remote_cluster_C2-C1 with settings {} [2018-10-18 01:18:21,027] - [rest_client] [140554369615616] - INFO - Replication created with id: cf3210709dd594147da343c8ee264a60/sasl_bucket_1/sasl_bucket_1 [2018-10-18 01:18:31,092] - [data_helper] [140554369615616] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:18:31,147] - [data_helper] [140554369615616] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:31,161] - [data_helper] [140554369615616] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:18:31,196] - [data_helper] [140554369615616] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:34,396] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:18:34,443] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:34,454] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:18:34,499] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:35,315] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:18:35,370] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:35,382] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:18:35,420] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:36,236] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:18:36,285] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:36,296] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:18:36,334] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:37,158] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 default [2018-10-18 01:18:37,198] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:37,208] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 default [2018-10-18 01:18:37,240] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:37,421] - [data_helper] [140554369615616] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:18:37,484] - [data_helper] [140554369615616] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:37,501] - [data_helper] [140554369615616] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:18:37,547] - [data_helper] [140554369615616] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:40,745] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:18:40,793] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:40,804] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:18:40,856] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:41,674] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:18:41,716] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:41,726] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:18:41,762] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:42,583] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:18:42,634] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:42,646] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:18:42,696] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:43,492] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.202:11210 sasl_bucket_1 [2018-10-18 01:18:43,536] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:43,548] - [data_helper] [140543115904768] - INFO - creating direct client 172.23.106.204:11210 sasl_bucket_1 [2018-10-18 01:18:43,585] - [data_helper] [140543115904768] - INFO - Atleast 1 of the server is on pre-spock version. Using the old ssl auth to connect to bucket. [2018-10-18 01:18:43,672] - [rest_client] [140554369615616] - INFO - Node versions in cluster [u'4.6.3-4136-enterprise', u'4.6.3-4136-enterprise'] [2018-10-18 01:18:43,692] - [rest_client] [140554369615616] - INFO - Node versions in cluster [u'6.5.0-1441-enterprise'] [2018-10-18 01:18:44,529] - [task] [140544013235968] - INFO - adding node 172.23.106.63:8091 to cluster [2018-10-18 01:18:44,529] - [rest_client] [140544013235968] - INFO - adding remote node @172.23.106.63:8091 to this cluster @172.23.106.202:8091 [2018-10-18 01:18:47,076] - [task] [140544013235968] - INFO - adding node 172.23.98.14:8091 to cluster [2018-10-18 01:18:47,077] - [rest_client] [140544013235968] - INFO - adding remote node @172.23.98.14:8091 to this cluster @172.23.106.202:8091 [2018-10-18 01:18:48,868] - [rest_client] [140544013235968] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.204,ns_1@172.23.98.14,ns_1@172.23.106.202,ns_1@172.23.106.63'} [2018-10-18 01:18:57,119] - [rest_client] [140544013235968] - INFO - rebalance operation started [2018-10-18 01:18:57,122] - [task] [140544013235968] - INFO - Rebalance - status: none, progress: 100.00% [2018-10-18 01:18:57,125] - [task] [140544013235968] - INFO - rebalancing was completed with progress: 100% in 0.00617694854736 sec [2018-10-18 01:18:57,125] - [upgradeXDCR] [140554369615616] - INFO - Rebalance in all 6.5.0-1441-enterprise nodes completed [2018-10-18 01:18:57,137] - [rest_client] [140554369615616] - INFO - Node 172.23.106.63 not part of cluster inactiveAdded [2018-10-18 01:18:57,138] - [rest_client] [140554369615616] - INFO - Node 172.23.98.14 not part of cluster inactiveAdded [2018-10-18 01:18:57,138] - [rest_client] [140554369615616] - INFO - Node versions in cluster [u'4.6.3-4136-enterprise', u'4.6.3-4136-enterprise'] [2018-10-18 01:18:57,139] - [basetestcase] [140554369615616] - INFO - sleep for 15 secs. ... [2018-10-18 01:19:12,164] - [rest_client] [140554369615616] - INFO - /diag/eval status on 172.23.106.202:8091: True content: 'ns_1@172.23.106.63' command: mb_master:master_node(). [2018-10-18 01:19:12,164] - [upgradeXDCR] [140554369615616] - INFO - after rebalance in the master is ns_1@172.23.106.63 [2018-10-18 01:19:12,165] - [upgradeXDCR] [140554369615616] - INFO - 6.5.0-1441-enterprise Node 172.23.106.63 becomes the master [2018-10-18 01:19:12,165] - [upgradeXDCR] [140554369615616] - INFO - Rebalancing out all old version nodes [2018-10-18 01:19:13,159] - [rest_client] [140544013235968] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.106.202,ns_1@172.23.106.204', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.106.204,ns_1@172.23.98.14,ns_1@172.23.106.202,ns_1@172.23.106.63'} [2018-10-18 01:19:13,167] - [rest_client] [140544013235968] - INFO - rebalance operation started [2018-10-18 01:19:13,173] - [rest_client] [140544013235968] - ERROR - {u'status': u'none', u'errorMessage': u'Please remove proxy ports from the following buckets: standard_bucket_1'} - rebalance failed [2018-10-18 01:19:13,203] - [rest_client] [140544013235968] - INFO - Latest logs from UI on 172.23.106.202: [2018-10-18 01:19:13,203] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.106.63', u'code': 0, u'text': u"Haven't heard from a higher priority node or a master, so I'm taking over.", u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:57.068Z', u'module': u'mb_master', u'tstamp': 1539850737068, u'type': u'info'} [2018-10-18 01:19:13,203] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 0, u'text': u'Service \'goxdcr\' exited with status 137. Restarting. Messages:\n2018-10-18T01:17:25.695-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":1304816,"TotalAlloc":1304816,"Sys":5867768,"Lookups":18,"Mallocs":14630,"Frees":628,"HeapAlloc":1304816,"HeapSys":2490368,"HeapIdle":401408,"HeapInuse":2088960,"HeapReleased":0,"HeapObjects":14002,"StackInuse":655360,"StackSys":655360,"MSpanInuse":35720,"MSpanSys":49152,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":1443357,"GCSys":169984,"OtherSys":1043163,"NextGC":4473924,"LastGC":0,"PauseTotalNs":0,"PauseNs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":0,"NumForcedGC":0,"GCCPUFraction":0,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":228,"Frees":0},{"Size":16,"Mallocs":3158,"Frees":0},{"Size":32,"Mallocs":5328,"Frees":0},{"Size":48,"Mallocs":1744,"Frees":0},{"Size":64,"Mallocs":655,"Frees":0},{"Size":80,"Mallocs":439,"Frees":0},{"Size":96,"Mallocs":558,"Frees":0},{"Size":112,"Mallocs":488,"Frees":0},{"Size":128,"Mallocs":151,"Frees":0},{"Size":144,"Mallocs":68,"Frees":0},{"Size":160,"Mallocs":73,"Frees":0},{"Size":176,"Mallocs":116,"Frees":0},{"Size":192,"Mallocs":8,"Frees":0},{"Size":208,"Mallocs":134,"Frees":0},{"Size":224,"Mallocs":75,"Frees":0},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":117,"Frees":0},{"Size":288,"Mallocs":50,"Frees":0},{"Size":320,"Mallocs":18,"Frees":0},{"Size":352,"Mallocs":270,"Frees":0},{"Size":384,"Mallocs":3,"Frees":0},{"Size":416,"Mallocs":80,"Frees":0},{"Size":448,"Mallocs":9,"Frees":0},{"Size":480,"Mallocs":4,"Frees":0},{"Size":512,"Mallocs":15,"Frees":0},{"Size":576,"Mallocs":17,"Frees":0},{"Size":640,"Mallocs":21,"Frees":0},{"Size":704,"Mallocs":25,"Frees":0},{"Size":768,"Mallocs":2,"Frees":0},{"Size":896,"Mallocs":12,"Frees":0},{"Size":1024,"Mallocs":13,"Frees":0},{"Size":1152,"Mallocs":15,"Frees":0},{"Size":1280,"Mallocs":12,"Frees":0},{"Size":1408,"Mallocs":1,"Frees":0},{"Size":1536,"Mallocs":2,"Frees":0},{"Size":1792,"Mallocs":7,"Frees":0},{"Size":2048,"Mallocs":22,"Frees":0},{"Size":2304,"Mallocs":9,"Frees":0},{"Size":2688,"Mallocs":7,"Frees":0},{"Size":3072,"Mallocs":2,"Frees":0},{"Size":3200,"Mallocs":1,"Frees":0},{"Size":3456,"Mallocs":5,"Frees":0},{"Size":4096,"Mallocs":20,"Frees":0},{"Size":4864,"Mallocs":8,"Frees":0},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":4,"Frees":0},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":1,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":2,"Frees":0},{"Size":9472,"Mallocs":0,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0},{"Size":19072,"Mallocs":0,"Frees":0}]}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:54.097Z', u'module': u'ns_log', u'tstamp': 1539850734097, u'type': u'info'} [2018-10-18 01:19:13,203] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 0, u'text': u'Service \'projector\' exited with status 137. Restarting. Messages:\n2018-10-18T01:16:43.885-07:00 [Info] memstats {"Alloc":1017760, "TotalAlloc":1521320, "Sys":4820992, "Lookups":11, "Mallocs":8372,"Frees":1544, "HeapAlloc":1017760, "HeapSys":2555904, "HeapIdle":712704, "HeapInuse":1843200,"HeapReleased":0, "HeapObjects":6828,"GCSys":313344, "LastGC":1539850543884852734,"PauseTotalNs":1350180, "PauseNs":[663606, 686574], "NumGC":2}\n2018-10-18T01:17:43.885-07:00 [Info] memstats {"Alloc":1019696, "TotalAlloc":1523256, "Sys":4820992, "Lookups":11, "Mallocs":8406,"Frees":1553, "HeapAlloc":1019696, "HeapSys":2555904, "HeapIdle":712704, "HeapInuse":1843200,"HeapReleased":0, "HeapObjects":6853,"GCSys":313344, "LastGC":1539850543884852734,"PauseTotalNs":1350180, "PauseNs":[], "NumGC":2}\n2018-10-18T01:18:43.885-07:00 [Info] memstats {"Alloc":829360, "TotalAlloc":1527192, "Sys":4820992, "Lookups":11, "Mallocs":8445,"Frees":2281, "HeapAlloc":829360, "HeapSys":2555904, "HeapIdle":974848, "HeapInuse":1581056,"HeapReleased":0, "HeapObjects":6164,"GCSys":313344, "LastGC":1539850663886345869,"PauseTotalNs":1500420, "PauseNs":[150240], "NumGC":3}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:54.096Z', u'module': u'ns_log', u'tstamp': 1539850734096, u'type': u'info'} [2018-10-18 01:19:13,203] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.106.63', u'code': 0, u'text': u'Service \'goxdcr\' exited with status 137. Restarting. Messages:\n2018-10-18T01:17:40.231-07:00 INFO GOXDCR.ReplMgr: Mem stats = {"Alloc":1309936,"TotalAlloc":1309936,"Sys":4427776,"Lookups":18,"Mallocs":14653,"Frees":628,"HeapAlloc":1309936,"HeapSys":2523136,"HeapIdle":409600,"HeapInuse":2113536,"HeapReleased":0,"HeapObjects":14025,"StackInuse":622592,"StackSys":622592,"MSpanInuse":36632,"MSpanSys":49152,"MCacheInuse":4800,"MCacheSys":16384,"BuckHashSys":2917,"GCSys":169984,"OtherSys":1043611,"NextGC":4473924,"LastGC":0,"PauseTotalNs":0,"PauseNs":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PauseEnd":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"NumGC":0,"NumForcedGC":0,"GCCPUFraction":0,"EnableGC":true,"DebugGC":false,"BySize":[{"Size":0,"Mallocs":0,"Frees":0},{"Size":8,"Mallocs":228,"Frees":0},{"Size":16,"Mallocs":3160,"Frees":0},{"Size":32,"Mallocs":5329,"Frees":0},{"Size":48,"Mallocs":1745,"Frees":0},{"Size":64,"Mallocs":659,"Frees":0},{"Size":80,"Mallocs":450,"Frees":0},{"Size":96,"Mallocs":563,"Frees":0},{"Size":112,"Mallocs":488,"Frees":0},{"Size":128,"Mallocs":151,"Frees":0},{"Size":144,"Mallocs":68,"Frees":0},{"Size":160,"Mallocs":72,"Frees":0},{"Size":176,"Mallocs":111,"Frees":0},{"Size":192,"Mallocs":8,"Frees":0},{"Size":208,"Mallocs":135,"Frees":0},{"Size":224,"Mallocs":75,"Frees":0},{"Size":240,"Mallocs":2,"Frees":0},{"Size":256,"Mallocs":117,"Frees":0},{"Size":288,"Mallocs":49,"Frees":0},{"Size":320,"Mallocs":18,"Frees":0},{"Size":352,"Mallocs":266,"Frees":0},{"Size":384,"Mallocs":4,"Frees":0},{"Size":416,"Mallocs":85,"Frees":0},{"Size":448,"Mallocs":11,"Frees":0},{"Size":480,"Mallocs":4,"Frees":0},{"Size":512,"Mallocs":14,"Frees":0},{"Size":576,"Mallocs":17,"Frees":0},{"Size":640,"Mallocs":21,"Frees":0},{"Size":704,"Mallocs":25,"Frees":0},{"Size":768,"Mallocs":3,"Frees":0},{"Size":896,"Mallocs":12,"Frees":0},{"Size":1024,"Mallocs":12,"Frees":0},{"Size":1152,"Mallocs":15,"Frees":0},{"Size":1280,"Mallocs":12,"Frees":0},{"Size":1408,"Mallocs":1,"Frees":0},{"Size":1536,"Mallocs":3,"Frees":0},{"Size":1792,"Mallocs":8,"Frees":0},{"Size":2048,"Mallocs":22,"Frees":0},{"Size":2304,"Mallocs":9,"Frees":0},{"Size":2688,"Mallocs":7,"Frees":0},{"Size":3072,"Mallocs":2,"Frees":0},{"Size":3200,"Mallocs":1,"Frees":0},{"Size":3456,"Mallocs":5,"Frees":0},{"Size":4096,"Mallocs":20,"Frees":0},{"Size":4864,"Mallocs":8,"Frees":0},{"Size":5376,"Mallocs":1,"Frees":0},{"Size":6144,"Mallocs":4,"Frees":0},{"Size":6528,"Mallocs":0,"Frees":0},{"Size":6784,"Mallocs":1,"Frees":0},{"Size":6912,"Mallocs":0,"Frees":0},{"Size":8192,"Mallocs":2,"Frees":0},{"Size":9472,"Mallocs":0,"Frees":0},{"Size":9728,"Mallocs":0,"Frees":0},{"Size":10240,"Mallocs":0,"Frees":0},{"Size":10880,"Mallocs":0,"Frees":0},{"Size":12288,"Mallocs":0,"Frees":0},{"Size":13568,"Mallocs":0,"Frees":0},{"Size":14336,"Mallocs":0,"Frees":0},{"Size":16384,"Mallocs":0,"Frees":0},{"Size":18432,"Mallocs":0,"Frees":0},{"Size":19072,"Mallocs":0,"Frees":0}]}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:51.557Z', u'module': u'ns_log', u'tstamp': 1539850731557, u'type': u'info'} [2018-10-18 01:19:13,203] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.106.63', u'code': 0, u'text': u'Service \'projector\' exited with status 137. Restarting. Messages:\n2018-10-18T01:15:46.944-07:00 [Info] PROJ[:9999] settings indexer.settings.inmemory_snapshot.interval will updated to `200`\n2018-10-18T01:15:46.944-07:00 [Info] PROJ[:9999] settings indexer.settings.moi.persistence_threads will updated to `2`\n2018-10-18T01:15:46.944-07:00 [Info] amd64 linux; cpus: 4; GOMAXPROCS: 4; version: go1.7.6\n2018-10-18T01:16:46.941-07:00 [Info] memstats {"Alloc":1024208, "TotalAlloc":1527640, "Sys":6260984, "Lookups":11, "Mallocs":8390,"Frees":1560, "HeapAlloc":1024208, "HeapSys":2621440, "HeapIdle":745472, "HeapInuse":1875968,"HeapReleased":0, "HeapObjects":6830,"GCSys":311296, "LastGC":1539850546940910871,"PauseTotalNs":2026079, "PauseNs":[1185023, 841056], "NumGC":2}\n2018-10-18T01:17:46.941-07:00 [Info] memstats {"Alloc":1026352, "TotalAlloc":1529784, "Sys":6260984, "Lookups":11, "Mallocs":8424,"Frees":1569, "HeapAlloc":1026352, "HeapSys":2621440, "HeapIdle":745472, "HeapInuse":1875968,"HeapReleased":0, "HeapObjects":6855,"GCSys":311296, "LastGC":1539850546940910871,"PauseTotalNs":2026079, "PauseNs":[], "NumGC":2}\n', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:51.555Z', u'module': u'ns_log', u'tstamp': 1539850731555, u'type': u'info'} [2018-10-18 01:19:13,204] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 0, u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"client_cert_auth">>,\n <<"collections_enabled">>,\n <<"datatype_snappy">>,\n <<"xattr_enabled">>]', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.904Z', u'module': u'memcached_config_mgr', u'tstamp': 1539850728904, u'type': u'info'} [2018-10-18 01:19:13,204] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 3, u'text': u'Node ns_1@172.23.98.14 joined cluster', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.830Z', u'module': u'ns_cluster', u'tstamp': 1539850728830, u'type': u'info'} [2018-10-18 01:19:13,204] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.98.14', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.98.14\'. Version: "6.5.0-1441-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2018-10-18T01:18:48.778Z', u'module': u'menelaus_sup', u'tstamp': 1539850728778, u'type': u'info'} [2018-10-18 01:19:13,204] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.106.202', u'code': 0, u'text': u'Conflicting configuration changes to field cbas_memory_quota:\n1166 and\n1185, choosing the former.', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.772Z', u'module': u'ns_config', u'tstamp': 1539850728772, u'type': u'info'} [2018-10-18 01:19:13,204] - [rest_client] [140544013235968] - ERROR - {u'node': u'ns_1@172.23.106.204', u'code': 0, u'text': u'Conflicting configuration changes to field cbas_memory_quota:\n1166 and\n1185, choosing the former.', u'shortText': u'message', u'serverTime': u'2018-10-18T01:18:48.771Z', u'module': u'ns_config', u'tstamp': 1539850728771, u'type': u'info'} [2018-10-18 01:19:13,226] - [xdcrnewbasetests] [140554369615616] - INFO - Collecting logs @ 172.23.106.202 [2018-10-18 01:19:13,231] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:19:13,434] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:19:14,255] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: /opt/couchbase/bin/cbcollect_info 172.23.106.202-20181018-0119-diag.zip [2018-10-18 01:21:45,729] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:21:46,123] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: rm -f /root/172.23.106.202-20181018-0119-diag.zip [2018-10-18 01:21:46,161] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:21:46,280] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:21:46,482] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.202 [2018-10-18 01:21:47,225] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: tar -zcvf 172.23.106.202-10182018-121-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-10-18 01:21:47,974] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:21:48,167] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.202: rm -f /root/172.23.106.202-10182018-121-couch.tar.gz [2018-10-18 01:21:48,205] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:21:48,319] - [xdcrnewbasetests] [140554369615616] - INFO - Collecting logs @ 172.23.106.204 [2018-10-18 01:21:48,321] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:21:48,454] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:21:48,957] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: /opt/couchbase/bin/cbcollect_info 172.23.106.204-20181018-0121-diag.zip [2018-10-18 01:23:52,088] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:23:52,310] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: rm -f /root/172.23.106.204-20181018-0121-diag.zip [2018-10-18 01:23:52,331] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:23:52,447] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:23:52,580] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.204 [2018-10-18 01:23:53,090] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: tar -zcvf 172.23.106.204-10182018-123-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-10-18 01:23:53,640] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:23:53,827] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.204: rm -f /root/172.23.106.204-10182018-123-couch.tar.gz [2018-10-18 01:23:53,848] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:23:53,962] - [xdcrnewbasetests] [140554369615616] - INFO - Collecting logs @ 172.23.106.208 [2018-10-18 01:23:53,964] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:23:54,061] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:23:54,334] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:23:54,732] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: /opt/couchbase/bin/cbcollect_info 172.23.106.208-20181018-0123-diag.zip [2018-10-18 01:25:32,005] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:25:32,191] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: rm -f /root/172.23.106.208-20181018-0123-diag.zip [2018-10-18 01:25:32,209] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:25:32,325] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:25:32,422] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.208 [2018-10-18 01:25:32,692] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:25:33,091] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: tar -zcvf 172.23.106.208-10182018-125-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-10-18 01:25:33,589] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:25:33,755] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.208: rm -f /root/172.23.106.208-10182018-125-couch.tar.gz [2018-10-18 01:25:33,773] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:25:33,887] - [xdcrnewbasetests] [140554369615616] - INFO - Collecting logs @ 172.23.106.209 [2018-10-18 01:25:33,889] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:25:34,055] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:25:34,604] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: /opt/couchbase/bin/cbcollect_info 172.23.106.209-20181018-0125-diag.zip [2018-10-18 01:27:59,709] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:27:59,953] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: rm -f /root/172.23.106.209-20181018-0125-diag.zip [2018-10-18 01:27:59,975] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:27:59,982] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:28:00,148] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.209 [2018-10-18 01:28:00,699] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: tar -zcvf 172.23.106.209-10182018-128-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-10-18 01:28:01,448] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:28:01,583] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.209: rm -f /root/172.23.106.209-10182018-128-couch.tar.gz [2018-10-18 01:28:01,620] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:28:01,734] - [xdcrnewbasetests] [140554369615616] - INFO - Collecting logs @ 172.23.106.63 [2018-10-18 01:28:01,736] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:28:01,902] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:28:02,427] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: /opt/couchbase/bin/cbcollect_info 172.23.106.63-20181018-0128-diag.zip [2018-10-18 01:29:15,382] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:29:15,586] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: rm -f /root/172.23.106.63-20181018-0128-diag.zip [2018-10-18 01:29:15,670] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:29:15,787] - [remote_util] [140554369615616] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:29:15,953] - [remote_util] [140554369615616] - INFO - Connected to 172.23.106.63 [2018-10-18 01:29:16,499] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: tar -zcvf 172.23.106.63-10182018-129-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-10-18 01:29:16,564] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:29:16,674] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.106.63: rm -f /root/172.23.106.63-10182018-129-couch.tar.gz [2018-10-18 01:29:16,695] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:29:16,809] - [xdcrnewbasetests] [140554369615616] - INFO - Collecting logs @ 172.23.98.14 [2018-10-18 01:29:16,811] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:29:16,908] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:29:17,198] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:29:17,607] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: /opt/couchbase/bin/cbcollect_info 172.23.98.14-20181018-0129-diag.zip [2018-10-18 01:30:24,896] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:30:25,144] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: rm -f /root/172.23.98.14-20181018-0129-diag.zip [2018-10-18 01:30:25,226] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:30:25,342] - [remote_util] [140554369615616] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:30:25,438] - [remote_util] [140554369615616] - INFO - Connected to 172.23.98.14 [2018-10-18 01:30:25,725] - [remote_util] [140554369615616] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:30:26,135] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: tar -zcvf 172.23.98.14-10182018-130-couch.tar.gz '/opt/couchbase/var/lib/couchbase/data' >/dev/null 2>&1 [2018-10-18 01:30:26,184] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:30:26,295] - [remote_util] [140554369615616] - INFO - running command.raw on 172.23.98.14: rm -f /root/172.23.98.14-10182018-130-couch.tar.gz [2018-10-18 01:30:26,368] - [remote_util] [140554369615616] - INFO - command executed successfully [2018-10-18 01:30:26,382] - [rest_client] [140554369615616] - INFO - Node 172.23.106.63 not part of cluster inactiveAdded [2018-10-18 01:30:26,383] - [rest_client] [140554369615616] - INFO - Node 172.23.98.14 not part of cluster inactiveAdded [2018-10-18 01:30:26,392] - [xdcrnewbasetests] [140554369615616] - WARNING - CLEANUP WAS SKIPPED [2018-10-18 01:30:26,395] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.202 with username:root password:couchbase ssh_key: [2018-10-18 01:30:26,597] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.202 [2018-10-18 01:30:27,441] - [remote_util] [140554426750720] - INFO - 172.23.106.202 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:30:27,585] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.204 with username:root password:couchbase ssh_key: [2018-10-18 01:30:27,750] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.204 [2018-10-18 01:30:28,331] - [remote_util] [140554426750720] - INFO - 172.23.106.204 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:30:28,470] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.208 with username:root password:couchbase ssh_key: [2018-10-18 01:30:28,566] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.208 [2018-10-18 01:30:28,841] - [remote_util] [140554426750720] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:30:29,316] - [remote_util] [140554426750720] - INFO - 172.23.106.208 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:30:29,372] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.209 with username:root password:couchbase ssh_key: [2018-10-18 01:30:29,538] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.209 [2018-10-18 01:30:30,136] - [remote_util] [140554426750720] - INFO - 172.23.106.209 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:30:30,235] - [remote_util] [140554426750720] - INFO - connecting to 172.23.106.63 with username:root password:couchbase ssh_key: [2018-10-18 01:30:30,424] - [remote_util] [140554426750720] - INFO - Connected to 172.23.106.63 [2018-10-18 01:30:31,035] - [remote_util] [140554426750720] - INFO - 172.23.106.63 **** The version file /opt/couchbase/ VERSION.txt exists [2018-10-18 01:30:31,127] - [remote_util] [140554426750720] - INFO - connecting to 172.23.98.14 with username:root password:couchbase ssh_key: [2018-10-18 01:30:31,224] - [remote_util] [140554426750720] - INFO - Connected to 172.23.98.14 [2018-10-18 01:30:31,471] - [remote_util] [140554426750720] - INFO - It could be other operating system. Go to check at other location [2018-10-18 01:30:31,911] - [remote_util] [140554426750720] - INFO - 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists