[2019-05-14 16:57:57,233] - [remote_util] [140244571428608] - INFO - connecting to 172.23.107.160 with username:root 
[2019-05-14 16:57:57,397] - [remote_util] [140244571428608] - INFO - Connected to 172.23.107.160
[2019-05-14 16:57:57,601] - [remote_util] [140244571428608] - INFO - It could be other operating system.  Go to check at other location
[2019-05-14 16:57:58,182] - [rest_client] [140244571428608] - INFO - Node version in cluster 6.5.0-3245-enterprise
[2019-05-14 16:57:58,194] - [rest_client] [140244571428608] - INFO - Node versions in cluster [u'6.5.0-3245-enterprise']
[2019-05-14 16:57:58,195] - [basetestcase] [140244571428608] - INFO - ==============  basetestcase setup was started for test #2 test_simple_ui_request==============
[2019-05-14 16:57:58,254] - [cluster_helper] [140244571428608] - INFO - waiting for ns_server @ 172.23.107.160:8091
[2019-05-14 16:57:58,260] - [cluster_helper] [140244571428608] - INFO - ns_server @ 172.23.107.160:8091 is running
[2019-05-14 16:57:58,261] - [basetestcase] [140244571428608] - INFO - initializing cluster
[2019-05-14 16:57:59,248] - [task] [140244563035904] - INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.160', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'20', 'mcdMemoryReserved': 6053, 'storageTotalRam': 7567, 'hostname': u'172.23.107.160:8091', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7f8d3c0cb8d0>], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3245-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'fts', u'index', u'kv', u'n1ql'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 7354159104, 'memoryTotal': 7934996480, 'memoryQuota': 3267, 'mcdMemoryAllocated': 6053, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
[2019-05-14 16:57:59,256] - [task] [140244563035904] - INFO - quota for index service will be 256 MB
[2019-05-14 16:57:59,256] - [task] [140244563035904] - INFO - set index quota to node 172.23.107.160 
[2019-05-14 16:57:59,256] - [rest_client] [140244563035904] - INFO - pools/default params : indexMemoryQuota=256
[2019-05-14 16:57:59,264] - [task] [140244563035904] - INFO - quota for fts service will be 512 MB
[2019-05-14 16:57:59,265] - [task] [140244563035904] - INFO - set both index and fts quota at node 172.23.107.160 
[2019-05-14 16:57:59,265] - [rest_client] [140244563035904] - INFO - pools/default params : ftsMemoryQuota=512
[2019-05-14 16:57:59,275] - [rest_client] [140244563035904] - INFO - pools/default params : memoryQuota=3267
[2019-05-14 16:57:59,285] - [rest_client] [140244563035904] - INFO - /node/controller/setupServices params on 172.23.107.160: 8091:services=kv%2Cindex%2Cn1ql%2Cfts&password=password&hostname=172.23.107.160&user=Administrator
[2019-05-14 16:57:59,291] - [rest_client] [140244563035904] - ERROR - POST http://172.23.107.160:8091//node/controller/setupServices body: services=kv%2Cindex%2Cn1ql%2Cfts&password=password&hostname=172.23.107.160&user=Administrator headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password
[2019-05-14 16:57:59,292] - [rest_client] [140244563035904] - INFO - This node is already provisioned with services, we do not consider this as failure for test case
[2019-05-14 16:57:59,292] - [rest_client] [140244563035904] - INFO - settings/web params on 172.23.107.160:8091:username=Administrator&password=password&port=8091
[2019-05-14 16:57:59,353] - [remote_util] [140244563035904] - INFO - connecting to 172.23.107.160 with username:root 
[2019-05-14 16:57:59,527] - [remote_util] [140244563035904] - INFO - Connected to 172.23.107.160
[2019-05-14 16:57:59,754] - [remote_util] [140244563035904] - INFO - It could be other operating system.  Go to check at other location
[2019-05-14 16:58:00,202] - [remote_util] [140244563035904] - INFO - running command.raw on 172.23.107.160: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).'
[2019-05-14 16:58:00,451] - [remote_util] [140244563035904] - INFO - command executed successfully
[2019-05-14 16:58:00,458] - [rest_client] [140244563035904] - INFO - /diag/eval status on 172.23.107.160:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version().
[2019-05-14 16:58:00,463] - [rest_client] [140244563035904] - INFO - /diag/eval status on 172.23.107.160:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version().
[2019-05-14 16:58:00,463] - [rest_client] [140244563035904] - INFO - settings/indexes params : storageMode=plasma
[2019-05-14 16:58:00,525] - [basetestcase] [140244571428608] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.107.160 ****
[2019-05-14 16:58:00,642] - [basetestcase] [140244571428608] - INFO - **** add 'admin' role to 'cbadminbucket' user ****
[2019-05-14 16:58:00,662] - [basetestcase] [140244571428608] - INFO - done initializing cluster
[2019-05-14 16:58:01,527] - [rest_client] [140244563035904] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@cb.local'}
[2019-05-14 16:58:01,540] - [rest_client] [140244563035904] - INFO - rebalance operation started
[2019-05-14 16:58:01,547] - [rest_client] [140244563035904] - INFO - rebalance percentage : 0.00 %
[2019-05-14 16:58:01,548] - [task] [140244563035904] - INFO - Rebalance - status: running, progress: 0.00%
[2019-05-14 16:58:11,574] - [rest_client] [140244563035904] - INFO - rebalance percentage : 0.00 %
[2019-05-14 16:58:11,574] - [task] [140244563035904] - INFO - Rebalance - status: running, progress: 0.00%
[2019-05-14 16:58:21,602] - [rest_client] [140244563035904] - INFO - rebalance percentage : 0.00 %
[2019-05-14 16:58:21,603] - [task] [140244563035904] - INFO - Rebalance - status: running, progress: 0.00%
[2019-05-14 16:58:31,628] - [rest_client] [140244563035904] - INFO - rebalance percentage : 0.00 %
[2019-05-14 16:58:31,628] - [task] [140244563035904] - INFO - Rebalance - status: running, progress: 0.00%
[2019-05-14 16:58:41,653] - [rest_client] [140244563035904] - INFO - rebalance percentage : 0.00 %
[2019-05-14 16:58:41,654] - [task] [140244563035904] - INFO - Rebalance - status: running, progress: 0.00%
[2019-05-14 16:58:51,681] - [rest_client] [140244563035904] - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed
[2019-05-14 16:58:51,690] - [rest_client] [140244563035904] - INFO - Latest logs from UI on 172.23.107.160:
[2019-05-14 16:58:51,691] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 0, u'text': u'Rebalance exited with reason {{badmatch,failed},\n                              [{ns_rebalancer,rebalance_body,5,\n                                   [{file,"src/ns_rebalancer.erl"},\n                                    {line,756}]},\n                               {async,\'-async_init/4-fun-2-\',3,\n                                   [{file,"src/async.erl"},{line,211}]}]}. Operation Id = cfba9a8393658e15b8ebf50932e28944', u'shortText': u'message', u'serverTime': u'2019-05-14T16:58:46.666Z', u'module': u'ns_orchestrator', u'tstamp': 1557878326666, u'type': u'critical'}
[2019-05-14 16:58:51,691] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 0, u'text': u"Starting rebalance, KeepNodes = ['ns_1@cb.local'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = cfba9a8393658e15b8ebf50932e28944", u'shortText': u'message', u'serverTime': u'2019-05-14T16:58:01.533Z', u'module': u'ns_orchestrator', u'tstamp': 1557878281533, u'type': u'info'}
[2019-05-14 16:58:51,691] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@cb.local\':\nUser-Agent:Python-httplib2/$Rev: 259 $\n2019-05-14 16:57:57.217311 : test_simple_ui_request finished \n', u'shortText': u'client-side error report', u'serverTime': u'2019-05-14T16:57:57.224Z', u'module': u'menelaus_web', u'tstamp': 1557878277224, u'type': u'warning'}
[2019-05-14 16:58:51,692] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@cb.local\':\nUser-Agent:Python-httplib2/$Rev: 259 $\n2019-05-14 16:57:55.073870 : test_simple_ui_request started \n', u'shortText': u'client-side error report', u'serverTime': u'2019-05-14T16:57:55.083Z', u'module': u'menelaus_web', u'tstamp': 1557878275083, u'type': u'warning'}
[2019-05-14 16:58:51,692] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 102, u'text': u'Client-side error-report for user "Administrator" on node \'ns_1@cb.local\':\nUser-Agent:Python-httplib2/$Rev: 259 $\n2019-05-14 16:57:51.549088 : test_simple_ui_request finished \n', u'shortText': u'client-side error report', u'serverTime': u'2019-05-14T16:57:51.557Z', u'module': u'menelaus_web', u'tstamp': 1557878271557, u'type': u'warning'}
[2019-05-14 16:58:51,692] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 0, u'text': u'Enabled auto-failover with timeout 120 and max count 1', u'shortText': u'message', u'serverTime': u'2019-05-14T16:57:41.554Z', u'module': u'auto_failover', u'tstamp': 1557878261554, u'type': u'info'}
[2019-05-14 16:58:51,693] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 0, u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"client_cert_auth">>,\n                                                                      <<"datatype_snappy">>,\n                                                                      <<"scramsha_fallback_salt">>,\n                                                                      <<"xattr_enabled">>]', u'shortText': u'message', u'serverTime': u'2019-05-14T16:57:41.550Z', u'module': u'memcached_config_mgr', u'tstamp': 1557878261550, u'type': u'info'}
[2019-05-14 16:58:51,693] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 0, u'text': u'Changed cluster compat mode from undefined to [6,5]', u'shortText': u'message', u'serverTime': u'2019-05-14T16:57:41.501Z', u'module': u'compat_mode_manager', u'tstamp': 1557878261501, u'type': u'warning'}
[2019-05-14 16:58:51,693] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 0, u'text': u"I'm the only node, so I'm the master.", u'shortText': u'message', u'serverTime': u'2019-05-14T16:57:41.428Z', u'module': u'mb_master', u'tstamp': 1557878261428, u'type': u'info'}
[2019-05-14 16:58:51,694] - [rest_client] [140244563035904] - ERROR - {u'node': u'ns_1@cb.local', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@cb.local\'. Version: "6.5.0-3245-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2019-05-14T16:57:41.238Z', u'module': u'menelaus_sup', u'tstamp': 1557878261238, u'type': u'info'}
[2019-05-14 16:58:51,699] - [remote_util] [140244818007872] - INFO - connecting to 172.23.107.160 with username:root 
[2019-05-14 16:58:51,863] - [remote_util] [140244818007872] - INFO - Connected to 172.23.107.160
[2019-05-14 16:58:52,062] - [remote_util] [140244818007872] - INFO - It could be other operating system.  Go to check at other location
[2019-05-14 16:58:52,506] - [remote_util] [140244818007872] - INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/cbcollect_info 172.23.107.160-20190514-1658-diag.zip 
[2019-05-14 17:00:31,205] - [remote_util] [140244818007872] - INFO - command executed successfully
[2019-05-14 17:00:31,490] - [remote_util] [140244818007872] - INFO - running command.raw on 172.23.107.160: rm -f /root/172.23.107.160-20190514-1658-diag.zip
[2019-05-14 17:00:31,572] - [remote_util] [140244818007872] - INFO - command executed successfully
[2019-05-14 17:00:31,688] - [remote_util] [140244818007872] - INFO - connecting to 172.23.107.160 with username:root 
[2019-05-14 17:00:31,852] - [remote_util] [140244818007872] - INFO - Connected to 172.23.107.160
[2019-05-14 17:00:32,058] - [remote_util] [140244818007872] - INFO - It could be other operating system.  Go to check at other location
[2019-05-14 17:00:32,486] - [remote_util] [140244818007872] - INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.160-20190514-1700-couch-dbinfo.txt
[2019-05-14 17:00:32,620] - [remote_util] [140244818007872] - INFO - command executed successfully
[2019-05-14 17:00:32,620] - [remote_util] [140244818007872] - INFO - Failed to open "/opt/couchbase/var/lib/couchbase/data/*/*[0-9]": no such file
[2019-05-14 17:00:32,656] - [remote_util] [140244818007872] - INFO - File 172.23.107.160-20190514-1700-couch-dbinfo.txt will be deleted
[2019-05-14 17:00:32,657] - [remote_util] [140244818007872] - INFO - running command.raw on 172.23.107.160: rm -rf /root/*172.23.107.160-20190514-1700-couch-dbinfo.txt*
[2019-05-14 17:00:32,676] - [remote_util] [140244818007872] - INFO - command executed successfully
[2019-05-14 17:00:32,676] - [remote_util] [140244818007872] - INFO - 172.23.107.160:sleep for 30 secs. ** Network or sever may be busy. **
Wait 30 seconds before executing next instrucion ...