[2019-07-15 10:17:40,603] - [remote_util] [139984642086656] - INFO - connecting to 172.23.107.160 with username:root [2019-07-15 10:17:40,784] - [remote_util] [139984642086656] - INFO - Connected to 172.23.107.160 [2019-07-15 10:17:40,998] - [remote_util] [139984642086656] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:17:41,507] - [rest_client] [139984642086656] - INFO - Node version in cluster 6.5.0-3763-enterprise [2019-07-15 10:17:41,520] - [rest_client] [139984642086656] - INFO - Node versions in cluster [u'6.5.0-3763-enterprise'] [2019-07-15 10:17:41,520] - [basetestcase] [139984642086656] - INFO - ============== basetestcase setup was started for test #5 rebalance_in_with_ops============== [2019-07-15 10:17:41,612] - [cluster_helper] [139984642086656] - INFO - waiting for ns_server @ 172.23.107.160:8091 [2019-07-15 10:17:41,618] - [cluster_helper] [139984642086656] - INFO - ns_server @ 172.23.107.160:8091 is running [2019-07-15 10:17:41,625] - [cluster_helper] [139984642086656] - INFO - waiting for ns_server @ 172.23.107.184:8091 [2019-07-15 10:17:41,631] - [cluster_helper] [139984642086656] - INFO - ns_server @ 172.23.107.184:8091 is running [2019-07-15 10:17:41,637] - [cluster_helper] [139984642086656] - INFO - waiting for ns_server @ 172.23.107.190:8091 [2019-07-15 10:17:41,643] - [cluster_helper] [139984642086656] - INFO - ns_server @ 172.23.107.190:8091 is running [2019-07-15 10:17:41,649] - [cluster_helper] [139984642086656] - INFO - waiting for ns_server @ 172.23.107.191:8091 [2019-07-15 10:17:41,655] - [cluster_helper] [139984642086656] - INFO - ns_server @ 172.23.107.191:8091 is running [2019-07-15 10:17:41,656] - [basetestcase] [139984642086656] - INFO - initializing cluster [2019-07-15 10:17:42,619] - [task] [139984633693952] - INFO - server: ip:172.23.107.160 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.160', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.160', 'uptime': u'909', 'mcdMemoryReserved': 6053, 'storageTotalRam': 7567, 'hostname': u'172.23.107.160:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3763-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'fts', u'index', u'kv', u'n1ql'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 6944759808, 'memoryTotal': 7934996480, 'memoryQuota': 4055, 'mcdMemoryAllocated': 6053, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-07-15 10:17:42,630] - [task] [139984633693952] - INFO - quota for index service will be 256 MB [2019-07-15 10:17:42,630] - [task] [139984633693952] - INFO - set index quota to node 172.23.107.160 [2019-07-15 10:17:42,630] - [rest_client] [139984633693952] - INFO - pools/default params : indexMemoryQuota=256 [2019-07-15 10:17:42,641] - [task] [139984633693952] - INFO - quota for fts service will be 512 MB [2019-07-15 10:17:42,641] - [task] [139984633693952] - INFO - set both index and fts quota at node 172.23.107.160 [2019-07-15 10:17:42,641] - [rest_client] [139984633693952] - INFO - pools/default params : ftsMemoryQuota=512 [2019-07-15 10:17:42,654] - [rest_client] [139984633693952] - INFO - pools/default params : memoryQuota=3287 [2019-07-15 10:17:42,666] - [rest_client] [139984633693952] - INFO - /node/controller/setupServices params on 172.23.107.160: 8091:services=kv%2Cindex%2Cn1ql%2Cfts&password=password&hostname=172.23.107.160&user=Administrator [2019-07-15 10:17:42,671] - [rest_client] [139984633693952] - ERROR - POST http://172.23.107.160:8091//node/controller/setupServices body: services=kv%2Cindex%2Cn1ql%2Cfts&password=password&hostname=172.23.107.160&user=Administrator headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password [2019-07-15 10:17:42,672] - [rest_client] [139984633693952] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2019-07-15 10:17:42,672] - [rest_client] [139984633693952] - INFO - settings/web params on 172.23.107.160:8091:username=Administrator&password=password&port=8091 [2019-07-15 10:17:42,740] - [remote_util] [139984633693952] - INFO - connecting to 172.23.107.160 with username:root [2019-07-15 10:17:42,940] - [remote_util] [139984633693952] - INFO - Connected to 172.23.107.160 [2019-07-15 10:17:43,167] - [remote_util] [139984633693952] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:17:43,608] - [remote_util] [139984633693952] - INFO - running command.raw on 172.23.107.160: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-07-15 10:17:43,856] - [remote_util] [139984633693952] - INFO - command executed successfully [2019-07-15 10:17:43,862] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.160:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:43,868] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.160:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:43,869] - [rest_client] [139984633693952] - INFO - settings/indexes params : storageMode=plasma [2019-07-15 10:17:43,939] - [task] [139984633693952] - INFO - server: ip:172.23.107.184 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.184', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.184', 'uptime': u'859', 'mcdMemoryReserved': 6250, 'storageTotalRam': 7812, 'hostname': u'172.23.107.184:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3763-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 7577829376, 'memoryTotal': 8192135168, 'memoryQuota': 4021, 'mcdMemoryAllocated': 6250, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-07-15 10:17:43,944] - [rest_client] [139984633693952] - ERROR - GET http://172.23.107.184:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-07-15 10:17:43,944] - [rest_client] [139984633693952] - INFO - pools/default params : memoryQuota=4187 [2019-07-15 10:17:43,950] - [rest_client] [139984633693952] - INFO - settings/web params on 172.23.107.184:8091:username=Administrator&password=password&port=8091 [2019-07-15 10:17:44,013] - [remote_util] [139984633693952] - INFO - connecting to 172.23.107.184 with username:root [2019-07-15 10:17:44,240] - [remote_util] [139984633693952] - INFO - Connected to 172.23.107.184 [2019-07-15 10:17:44,678] - [remote_util] [139984633693952] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:17:45,151] - [remote_util] [139984633693952] - INFO - running command.raw on 172.23.107.184: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-07-15 10:17:45,216] - [remote_util] [139984633693952] - INFO - command executed successfully [2019-07-15 10:17:45,221] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.184:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:45,225] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.184:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:45,225] - [rest_client] [139984633693952] - INFO - settings/indexes params : storageMode=plasma [2019-07-15 10:17:45,368] - [task] [139984633693952] - INFO - server: ip:172.23.107.190 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.190', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.190', 'uptime': u'939', 'mcdMemoryReserved': 6053, 'storageTotalRam': 7567, 'hostname': u'172.23.107.190:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3763-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'fts', u'index', u'kv', u'n1ql'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 6938320896, 'memoryTotal': 7934996480, 'memoryQuota': 4055, 'mcdMemoryAllocated': 6053, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-07-15 10:17:45,375] - [rest_client] [139984633693952] - INFO - pools/default params : memoryQuota=4055 [2019-07-15 10:17:45,383] - [rest_client] [139984633693952] - INFO - settings/web params on 172.23.107.190:8091:username=Administrator&password=password&port=8091 [2019-07-15 10:17:45,443] - [remote_util] [139984633693952] - INFO - connecting to 172.23.107.190 with username:root [2019-07-15 10:17:45,642] - [remote_util] [139984633693952] - INFO - Connected to 172.23.107.190 [2019-07-15 10:17:45,857] - [remote_util] [139984633693952] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:17:46,307] - [remote_util] [139984633693952] - INFO - running command.raw on 172.23.107.190: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-07-15 10:17:46,556] - [remote_util] [139984633693952] - INFO - command executed successfully [2019-07-15 10:17:46,561] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.190:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:46,565] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.190:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:46,565] - [rest_client] [139984633693952] - INFO - settings/indexes params : storageMode=plasma [2019-07-15 10:17:46,620] - [task] [139984633693952] - INFO - server: ip:172.23.107.191 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.191', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.107.191', 'uptime': u'924', 'mcdMemoryReserved': 6053, 'storageTotalRam': 7567, 'hostname': u'172.23.107.191:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-3763-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 7351234560, 'memoryTotal': 7934996480, 'memoryQuota': 3874, 'mcdMemoryAllocated': 6053, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-07-15 10:17:46,624] - [rest_client] [139984633693952] - ERROR - GET http://172.23.107.191:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-07-15 10:17:46,625] - [rest_client] [139984633693952] - INFO - pools/default params : memoryQuota=4055 [2019-07-15 10:17:46,631] - [rest_client] [139984633693952] - INFO - settings/web params on 172.23.107.191:8091:username=Administrator&password=password&port=8091 [2019-07-15 10:17:46,692] - [remote_util] [139984633693952] - INFO - connecting to 172.23.107.191 with username:root [2019-07-15 10:17:46,869] - [remote_util] [139984633693952] - INFO - Connected to 172.23.107.191 [2019-07-15 10:17:47,071] - [remote_util] [139984633693952] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:17:47,513] - [remote_util] [139984633693952] - INFO - running command.raw on 172.23.107.191: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-07-15 10:17:47,745] - [remote_util] [139984633693952] - INFO - command executed successfully [2019-07-15 10:17:47,749] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.191:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:47,753] - [rest_client] [139984633693952] - INFO - /diag/eval status on 172.23.107.191:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-07-15 10:17:47,753] - [rest_client] [139984633693952] - INFO - settings/indexes params : storageMode=plasma [2019-07-15 10:17:47,813] - [basetestcase] [139984642086656] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.107.160 **** [2019-07-15 10:17:47,832] - [rest_client] [139984642086656] - ERROR - DELETE http://172.23.107.160:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-07-15 10:17:47,833] - [internal_user] [139984642086656] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-07-15 10:17:47,914] - [basetestcase] [139984642086656] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2019-07-15 10:17:47,933] - [basetestcase] [139984642086656] - INFO - done initializing cluster [2019-07-15 10:17:47,935] - [remote_util] [139984642086656] - INFO - connecting to 172.23.107.160 with username:root [2019-07-15 10:17:48,112] - [remote_util] [139984642086656] - INFO - Connected to 172.23.107.160 [2019-07-15 10:17:48,344] - [remote_util] [139984642086656] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:17:48,780] - [remote_util] [139984642086656] - INFO - running command.raw on 172.23.107.160: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-07-15 10:17:49,028] - [remote_util] [139984642086656] - INFO - command executed successfully [2019-07-15 10:17:49,028] - [basetestcase] [139984642086656] - INFO - Enabled diag/eval for non-local hosts from 172.23.107.160 [2019-07-15 10:17:49,834] - [rest_client] [139984633693952] - INFO - http://172.23.107.160:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=3287&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 [2019-07-15 10:17:49,847] - [rest_client] [139984633693952] - INFO - 0.01 seconds to create bucket default [2019-07-15 10:17:49,847] - [bucket_helper] [139984633693952] - INFO - waiting for memcached bucket : default in 172.23.107.160 to accept set ops [2019-07-15 10:17:50,532] - [data_helper] [139984633693952] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:50,643] - [data_helper] [139984633693952] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:52,764] - [task] [139984633693952] - INFO - bucket 'default' was created with per node RAM quota: 3287 [2019-07-15 10:17:52,786] - [basetestcase] [139984642086656] - INFO - ============== basetestcase setup was finished for test #5 rebalance_in_with_ops ============== [2019-07-15 10:17:52,813] - [basetestcase] [139984642086656] - INFO - ------- Cluster statistics ------- [2019-07-15 10:17:52,813] - [basetestcase] [139984642086656] - INFO - 172.23.107.160:8091 => {'swap_mem_used': 25128960, 'cpu_utilization': 8.808290155440414, 'mem_free': 6937800704, 'swap_mem_total': 3758092288, 'mem_total': 7934996480, 'services': [u'fts', u'index', u'kv', u'n1ql']} [2019-07-15 10:17:52,813] - [basetestcase] [139984642086656] - INFO - --- End of cluster statistics --- [2019-07-15 10:17:52,878] - [data_helper] [139984642086656] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:53,851] - [data_helper] [139984616908544] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:53,855] - [data_helper] [139984616908544] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:53,855] - [data_helper] [139984616908544] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:53,860] - [data_helper] [139984616908544] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:54,148] - [basetestcase] [139984642086656] - INFO - Servers to check bucket-seqno: [ip:172.23.107.160 port:8091 ssh_username:root] [2019-07-15 10:17:54,148] - [basetestcase] [139984642086656] - INFO - Record vbucket for the bucket default [2019-07-15 10:17:54,197] - [data_helper] [139984642086656] - INFO - creating direct client 172.23.107.160:11210 default [2019-07-15 10:17:54,767] - [task] [139984633693952] - INFO - adding node 172.23.107.184:8091 to cluster [2019-07-15 10:17:54,768] - [rest_client] [139984633693952] - INFO - adding remote node @172.23.107.184:8091 to this cluster @172.23.107.160:8091 [2019-07-15 10:17:57,516] - [task] [139984633693952] - INFO - adding node 172.23.107.190:8091 to cluster [2019-07-15 10:17:57,516] - [rest_client] [139984633693952] - INFO - adding remote node @172.23.107.190:8091 to this cluster @172.23.107.160:8091 [2019-07-15 10:18:00,121] - [task] [139984633693952] - INFO - adding node 172.23.107.191:8091 to cluster [2019-07-15 10:18:00,121] - [rest_client] [139984633693952] - INFO - adding remote node @172.23.107.191:8091 to this cluster @172.23.107.160:8091 [2019-07-15 10:18:02,820] - [rest_client] [139984633693952] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.107.160,ns_1@172.23.107.190,ns_1@172.23.107.184,ns_1@172.23.107.191'} [2019-07-15 10:18:02,832] - [rest_client] [139984633693952] - INFO - rebalance operation started [2019-07-15 10:18:02,836] - [rest_client] [139984633693952] - INFO - rebalance percentage : 0.00 % [2019-07-15 10:18:02,836] - [task] [139984633693952] - INFO - Rebalance - status: running, progress: 0.00% [2019-07-15 10:18:12,869] - [rest_client] [139984633693952] - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2019-07-15 10:18:12,967] - [rest_client] [139984633693952] - INFO - Latest logs from UI on 172.23.107.160: [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.160', u'code': 0, u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.14657.1>,\n {{dcp_wait_for_data_move_failed,"default",\n 1022,\'ns_1@172.23.107.160\',\n [\'ns_1@172.23.107.191\',\n \'ns_1@172.23.107.190\'],\n {error,no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,465}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,232}]}]}}}}.\nRebalance Operation Id = 28925993fb4e1742245387a9d86c32d6', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:05.523Z', u'module': u'ns_orchestrator', u'tstamp': 1563211085523, u'type': u'critical'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.160', u'code': 0, u'text': u'Worker <0.14569.1> (for action {move,{1022,\n [\'ns_1@172.23.107.160\',undefined],\n [\'ns_1@172.23.107.191\',\n \'ns_1@172.23.107.190\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.14657.1>,\n {{dcp_wait_for_data_move_failed,\n "default",\n 1022,\n \'ns_1@172.23.107.160\',\n [\'ns_1@172.23.107.191\',\n \'ns_1@172.23.107.190\'],\n {error,\n no_stats_for_this_vbucket}},\n [{ns_single_vbucket_mover,\n \'-wait_dcp_data_move/5-fun-0-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 465}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 232}]}]}}}', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:05.519Z', u'module': u'ns_vbucket_mover', u'tstamp': 1563211085519, u'type': u'critical'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.160', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:05.372Z', u'module': u'ns_vbucket_mover', u'tstamp': 1563211085372, u'type': u'info'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.190', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.190\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:03.735Z', u'module': u'ns_memcached', u'tstamp': 1563211083735, u'type': u'info'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.184', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.184\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:03.725Z', u'module': u'ns_memcached', u'tstamp': 1563211083725, u'type': u'info'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.191', u'code': 0, u'text': u'Bucket "default" loaded on node \'ns_1@172.23.107.191\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:03.722Z', u'module': u'ns_memcached', u'tstamp': 1563211083722, u'type': u'info'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.160', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:03.586Z', u'module': u'ns_rebalancer', u'tstamp': 1563211083586, u'type': u'info'} [2019-07-15 10:18:12,968] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.191', u'code': 0, u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:02.842Z', u'module': u'memcached_config_mgr', u'tstamp': 1563211082842, u'type': u'info'} [2019-07-15 10:18:12,969] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.160', u'code': 0, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.107.160','ns_1@172.23.107.190',\n 'ns_1@172.23.107.184','ns_1@172.23.107.191'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 28925993fb4e1742245387a9d86c32d6", u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:02.824Z', u'module': u'ns_orchestrator', u'tstamp': 1563211082824, u'type': u'info'} [2019-07-15 10:18:12,969] - [rest_client] [139984633693952] - ERROR - {u'node': u'ns_1@172.23.107.191', u'code': 3, u'text': u'Node ns_1@172.23.107.191 joined cluster', u'shortText': u'message', u'serverTime': u'2019-07-15T10:18:02.788Z', u'module': u'ns_cluster', u'tstamp': 1563211082788, u'type': u'info'} [2019-07-15 10:18:12,993] - [basetestcase] [139984642086656] - INFO - ------- Cluster statistics ------- [2019-07-15 10:18:12,994] - [basetestcase] [139984642086656] - INFO - 172.23.107.160:8091 => {'swap_mem_used': 25128960, 'cpu_utilization': 2.864583333333333, 'mem_free': 6788976640, 'swap_mem_total': 3758092288, 'mem_total': 7934996480, 'services': [u'fts', u'index', u'kv', u'n1ql']} [2019-07-15 10:18:12,994] - [basetestcase] [139984642086656] - INFO - 172.23.107.190:8091 => {'swap_mem_used': 31162368, 'cpu_utilization': 1.503759398496241, 'mem_free': 7222910976, 'swap_mem_total': 3758092288, 'mem_total': 7934996480, 'services': [u'kv']} [2019-07-15 10:18:12,995] - [basetestcase] [139984642086656] - INFO - 172.23.107.184:8091 => {'swap_mem_used': 19656704, 'cpu_utilization': 1.758793969849246, 'mem_free': 7518093312, 'swap_mem_total': 3758092288, 'mem_total': 8192135168, 'services': [u'kv']} [2019-07-15 10:18:12,995] - [basetestcase] [139984642086656] - INFO - 172.23.107.191:8091 => {'swap_mem_used': 26513408, 'cpu_utilization': 1.256281407035176, 'mem_free': 7284224000, 'swap_mem_total': 3758092288, 'mem_total': 7934996480, 'services': [u'kv']} [2019-07-15 10:18:12,996] - [basetestcase] [139984642086656] - INFO - --- End of cluster statistics --- [2019-07-15 10:18:12,999] - [remote_util] [139984642086656] - INFO - connecting to 172.23.107.160 with username:root [2019-07-15 10:18:13,176] - [remote_util] [139984642086656] - INFO - Connected to 172.23.107.160 [2019-07-15 10:18:13,405] - [remote_util] [139984642086656] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:18:14,061] - [remote_util] [139984642086656] - INFO - connecting to 172.23.107.184 with username:root [2019-07-15 10:18:14,294] - [remote_util] [139984642086656] - INFO - Connected to 172.23.107.184 [2019-07-15 10:18:14,731] - [remote_util] [139984642086656] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:18:15,472] - [remote_util] [139984642086656] - INFO - connecting to 172.23.107.190 with username:root [2019-07-15 10:18:15,650] - [remote_util] [139984642086656] - INFO - Connected to 172.23.107.190 [2019-07-15 10:18:15,842] - [remote_util] [139984642086656] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:18:16,497] - [remote_util] [139984642086656] - INFO - connecting to 172.23.107.191 with username:root [2019-07-15 10:18:16,675] - [remote_util] [139984642086656] - INFO - Connected to 172.23.107.191 [2019-07-15 10:18:16,868] - [remote_util] [139984642086656] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:18:17,527] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.160 with username:root [2019-07-15 10:18:17,706] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.160 [2019-07-15 10:18:17,941] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:18:18,372] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/cbcollect_info 172.23.107.160-20190715-1018-diag.zip [2019-07-15 10:20:45,539] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:20:45,854] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.160: rm -f /root/172.23.107.160-20190715-1018-diag.zip [2019-07-15 10:20:45,891] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:20:46,007] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.184 with username:root [2019-07-15 10:20:46,244] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.184 [2019-07-15 10:20:46,664] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:20:47,102] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/cbcollect_info 172.23.107.184-20190715-1020-diag.zip [2019-07-15 10:22:22,499] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:22:22,953] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.184: rm -f /root/172.23.107.184-20190715-1020-diag.zip [2019-07-15 10:22:22,990] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:22:23,106] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.190 with username:root [2019-07-15 10:22:23,283] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.190 [2019-07-15 10:22:23,473] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:22:23,903] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.190: /opt/couchbase/bin/cbcollect_info 172.23.107.190-20190715-1022-diag.zip [2019-07-15 10:24:02,529] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:24:03,075] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.190: rm -f /root/172.23.107.190-20190715-1022-diag.zip [2019-07-15 10:24:03,095] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:24:03,211] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.191 with username:root [2019-07-15 10:24:03,411] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.191 [2019-07-15 10:24:03,600] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:24:04,040] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.191: /opt/couchbase/bin/cbcollect_info 172.23.107.191-20190715-1024-diag.zip [2019-07-15 10:25:38,779] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:39,201] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.191: rm -f /root/172.23.107.191-20190715-1024-diag.zip [2019-07-15 10:25:39,222] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:39,338] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.160 with username:root [2019-07-15 10:25:39,524] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.160 [2019-07-15 10:25:39,764] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:25:40,208] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.160: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.160-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:40,392] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:40,605] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.160: rm -f /root/172.23.107.160-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:40,623] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:40,739] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.184 with username:root [2019-07-15 10:25:40,970] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.184 [2019-07-15 10:25:41,418] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:25:41,869] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.184: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.184-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:41,954] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:42,207] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.184: rm -f /root/172.23.107.184-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:42,226] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:42,341] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.190 with username:root [2019-07-15 10:25:42,521] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.190 [2019-07-15 10:25:42,711] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:25:43,146] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.190: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.190-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:43,279] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:43,473] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.190: rm -f /root/172.23.107.190-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:43,492] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:43,607] - [remote_util] [139984908789568] - INFO - connecting to 172.23.107.191 with username:root [2019-07-15 10:25:43,786] - [remote_util] [139984908789568] - INFO - Connected to 172.23.107.191 [2019-07-15 10:25:44,009] - [remote_util] [139984908789568] - INFO - It could be other operating system. Go to check at other location [2019-07-15 10:25:44,431] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.191: /opt/couchbase/bin/couch_dbinfo -i /opt/couchbase/var/lib/couchbase/data/*/*[0-9] >172.23.107.191-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:44,564] - [remote_util] [139984908789568] - INFO - command executed successfully [2019-07-15 10:25:44,799] - [remote_util] [139984908789568] - INFO - running command.raw on 172.23.107.191: rm -f /root/172.23.107.191-20190715-1025-couch-dbinfo.txt [2019-07-15 10:25:44,881] - [remote_util] [139984908789568] - INFO - command executed successfully