[2020-06-04 06:52:10,471] - [eventing_base] [139642271692544] - INFO - Starting Test: test_eventing_rebalance_with_multiple_kv_nodes [2020-06-04 06:52:10,477] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:52:10,614] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 06:52:10,929] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:52:11,455] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:52:11,575] - [rest_client] [139642271692544] - INFO - Node version in cluster 6.6.0-7737-enterprise [2020-06-04 06:52:11,615] - [rest_client] [139642271692544] - INFO - Node versions in cluster [u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise'] [2020-06-04 06:52:11,616] - [basetestcase] [139642271692544] - INFO - ============== basetestcase setup was started for test #8 test_eventing_rebalance_with_multiple_kv_nodes============== [2020-06-04 06:52:11,756] - [bucket_helper] [139642271692544] - INFO - deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket'] on 172.23.105.10 [2020-06-04 06:52:11,757] - [bucket_helper] [139642271692544] - INFO - remove bucket dst_bucket ... [2020-06-04 06:52:12,518] - [bucket_helper] [139642271692544] - INFO - deleted bucket : dst_bucket from 172.23.105.10 [2020-06-04 06:52:12,519] - [bucket_helper] [139642271692544] - INFO - waiting for bucket deletion to complete.... [2020-06-04 06:52:12,564] - [rest_client] [139642271692544] - INFO - node 172.23.105.10 existing buckets : [u'metadata', u'src_bucket'] [2020-06-04 06:52:12,566] - [bucket_helper] [139642271692544] - INFO - remove bucket metadata ... [2020-06-04 06:52:13,318] - [bucket_helper] [139642271692544] - INFO - deleted bucket : metadata from 172.23.105.10 [2020-06-04 06:52:13,319] - [bucket_helper] [139642271692544] - INFO - waiting for bucket deletion to complete.... [2020-06-04 06:52:13,354] - [rest_client] [139642271692544] - INFO - node 172.23.105.10 existing buckets : [u'src_bucket'] [2020-06-04 06:52:13,355] - [bucket_helper] [139642271692544] - INFO - remove bucket src_bucket ... [2020-06-04 06:52:15,092] - [bucket_helper] [139642271692544] - INFO - deleted bucket : src_bucket from 172.23.105.10 [2020-06-04 06:52:15,092] - [bucket_helper] [139642271692544] - INFO - waiting for bucket deletion to complete.... [2020-06-04 06:52:15,116] - [rest_client] [139642271692544] - INFO - node 172.23.105.10 existing buckets : [] [2020-06-04 06:52:15,117] - [bucket_helper] [139642271692544] - INFO - sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket']) were deleted completely. [2020-06-04 06:52:17,389] - [cluster_helper] [139642271692544] - INFO - rebalancing all nodes in order to remove nodes [2020-06-04 06:52:17,400] - [rest_client] [139642271692544] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.188,ns_1@172.23.105.137,ns_1@172.23.105.122,ns_1@172.23.105.163,ns_1@172.23.105.185', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.188,ns_1@172.23.105.10,ns_1@172.23.105.137,ns_1@172.23.105.122,ns_1@172.23.105.163,ns_1@172.23.105.185'} [2020-06-04 06:52:17,424] - [rest_client] [139642271692544] - INFO - rebalance operation started [2020-06-04 06:52:17,436] - [rest_client] [139642271692544] - INFO - rebalance percentage : 0.00 % [2020-06-04 06:52:27,457] - [rest_client] [139642271692544] - INFO - rebalance percentage : 21.43 % [2020-06-04 06:52:37,479] - [rest_client] [139642271692544] - INFO - rebalance percentage : 92.86 % [2020-06-04 06:52:57,522] - [rest_client] [139642271692544] - INFO - rebalance progress took 40.10 seconds [2020-06-04 06:52:57,523] - [rest_client] [139642271692544] - INFO - sleep for 10 seconds after rebalance... [2020-06-04 06:53:07,700] - [cluster_helper] [139642271692544] - INFO - removed all the nodes from cluster associated with ip:172.23.105.10 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.130', 8091), (u'ns_1@172.23.105.188', 8091), (u'ns_1@172.23.105.137', 8091), (u'ns_1@172.23.105.122', 8091), (u'ns_1@172.23.105.163', 8091), (u'ns_1@172.23.105.185', 8091)] [2020-06-04 06:53:07,714] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.10:8091 [2020-06-04 06:53:07,739] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.10:8091 is running [2020-06-04 06:53:07,751] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.122:8091 [2020-06-04 06:53:07,763] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.122:8091 is running [2020-06-04 06:53:07,775] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.130:8091 [2020-06-04 06:53:07,787] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.130:8091 is running [2020-06-04 06:53:07,805] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.137:8091 [2020-06-04 06:53:07,835] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.137:8091 is running [2020-06-04 06:53:07,848] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.163:8091 [2020-06-04 06:53:07,860] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.163:8091 is running [2020-06-04 06:53:07,872] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.185:8091 [2020-06-04 06:53:07,884] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.185:8091 is running [2020-06-04 06:53:07,900] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.105.188:8091 [2020-06-04 06:53:07,915] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.105.188:8091 is running [2020-06-04 06:53:07,940] - [cluster_helper] [139642271692544] - INFO - waiting for ns_server @ 172.23.108.238:8091 [2020-06-04 06:53:07,952] - [cluster_helper] [139642271692544] - INFO - ns_server @ 172.23.108.238:8091 is running [2020-06-04 06:53:07,953] - [basetestcase] [139642271692544] - INFO - initializing cluster [2020-06-04 06:53:07,957] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:53:08,056] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 06:53:08,370] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:08,867] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:08,897] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:53:08,997] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 06:53:09,270] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:09,703] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:09,790] - [remote_util] [139642271692544] - INFO - 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:10,047] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:10,047] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: systemctl stop couchbase-server.service [2020-06-04 06:53:11,771] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:11,771] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:53:11,836] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:53:11,862] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:11,862] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:53:11,948] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:11,952] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:53:12,050] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 06:53:12,329] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:12,774] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:12,861] - [remote_util] [139642271692544] - INFO - 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:12,862] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:12,972] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:53:13,172] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:13,172] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: systemctl start couchbase-server.service [2020-06-04 06:53:13,209] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:13,209] - [remote_util] [139642271692544] - INFO - 172.23.105.10:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:53:18,215] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:53:18,252] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:18,253] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 14924 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:53:18,254] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:18,328] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:53:18,454] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 06:53:18,557] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.122 [2020-06-04 06:53:18,822] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:19,238] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:19,282] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 06:53:19,381] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.122 [2020-06-04 06:53:19,615] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:20,029] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:20,111] - [remote_util] [139642271692544] - INFO - 172.23.105.122 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:20,366] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:20,367] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.122: systemctl stop couchbase-server.service [2020-06-04 06:53:21,841] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:21,841] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:53:21,956] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.122: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:53:21,984] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:21,984] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.122: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:53:22,067] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:22,074] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 06:53:22,173] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.122 [2020-06-04 06:53:22,410] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:22,882] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:22,974] - [remote_util] [139642271692544] - INFO - 172.23.105.122 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:22,975] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:23,077] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:53:23,329] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:23,330] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.122: systemctl start couchbase-server.service [2020-06-04 06:53:23,351] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:23,351] - [remote_util] [139642271692544] - INFO - 172.23.105.122:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:53:28,357] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.122: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:53:28,393] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:28,394] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 15906 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:53:28,394] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:28,451] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:53:28,572] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 06:53:28,710] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.130 [2020-06-04 06:53:29,000] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:29,499] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:29,527] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 06:53:29,626] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.130 [2020-06-04 06:53:29,968] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:30,480] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:30,578] - [remote_util] [139642271692544] - INFO - 172.23.105.130 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:30,837] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:30,837] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.130: systemctl stop couchbase-server.service [2020-06-04 06:53:32,411] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:32,411] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:53:32,526] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:53:32,546] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:32,546] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:53:32,629] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:32,633] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 06:53:32,732] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.130 [2020-06-04 06:53:33,049] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:33,522] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:33,609] - [remote_util] [139642271692544] - INFO - 172.23.105.130 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:33,610] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:33,718] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:53:33,995] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:34,000] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.130: systemctl start couchbase-server.service [2020-06-04 06:53:34,038] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:34,038] - [remote_util] [139642271692544] - INFO - 172.23.105.130:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:53:39,044] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:53:39,081] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:39,081] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 91579 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:53:39,082] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:39,183] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:53:39,313] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 06:53:39,449] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.137 [2020-06-04 06:53:39,807] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:40,274] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:40,326] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 06:53:40,464] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.137 [2020-06-04 06:53:40,796] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:41,478] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:41,571] - [remote_util] [139642271692544] - INFO - 172.23.105.137 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:41,674] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:41,674] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: systemctl stop couchbase-server.service [2020-06-04 06:53:43,248] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:43,248] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:53:43,364] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:53:43,388] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:43,388] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:53:43,473] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:43,477] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 06:53:43,579] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.137 [2020-06-04 06:53:43,922] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:44,389] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:44,479] - [remote_util] [139642271692544] - INFO - 172.23.105.137 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:44,480] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:44,590] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:53:44,731] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:44,732] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: systemctl start couchbase-server.service [2020-06-04 06:53:44,772] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:44,772] - [remote_util] [139642271692544] - INFO - 172.23.105.137:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:53:49,777] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:53:49,815] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:49,816] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 6377 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:53:49,816] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:49,919] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:53:50,043] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 06:53:50,142] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.163 [2020-06-04 06:53:50,419] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:50,901] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:50,944] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 06:53:51,043] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.163 [2020-06-04 06:53:51,276] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:51,620] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:51,705] - [remote_util] [139642271692544] - INFO - 172.23.105.163 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:51,951] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:51,951] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.163: systemctl stop couchbase-server.service [2020-06-04 06:53:53,374] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:53,375] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:53:53,490] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.163: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:53:53,509] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:53,510] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.163: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:53:53,584] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:53,590] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 06:53:53,688] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.163 [2020-06-04 06:53:53,979] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:53:54,431] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:53:54,484] - [remote_util] [139642271692544] - INFO - 172.23.105.163 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:53:54,485] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:54,588] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:53:54,811] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:53:54,811] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.163: systemctl start couchbase-server.service [2020-06-04 06:53:54,833] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:54,834] - [remote_util] [139642271692544] - INFO - 172.23.105.163:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:53:59,840] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.163: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:53:59,864] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:53:59,865] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 6359 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:53:59,865] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:53:59,965] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:54:00,084] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 06:54:00,183] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.185 [2020-06-04 06:54:00,485] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:00,951] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:00,988] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 06:54:01,087] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.185 [2020-06-04 06:54:01,316] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:01,754] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:01,809] - [remote_util] [139642271692544] - INFO - 172.23.105.185 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:54:02,065] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:54:02,066] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.185: systemctl stop couchbase-server.service [2020-06-04 06:54:03,541] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:03,541] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:54:03,656] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.185: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:54:03,671] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:03,672] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.185: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:54:03,758] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:03,773] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 06:54:03,873] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.185 [2020-06-04 06:54:04,113] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:04,562] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:04,624] - [remote_util] [139642271692544] - INFO - 172.23.105.185 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:54:04,624] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:54:04,727] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:54:04,980] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:54:04,980] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.185: systemctl start couchbase-server.service [2020-06-04 06:54:05,003] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:05,003] - [remote_util] [139642271692544] - INFO - 172.23.105.185:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:54:10,006] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.185: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:54:10,044] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:10,044] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 18979 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:54:10,044] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:54:10,100] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:54:10,219] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 06:54:10,523] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.188 [2020-06-04 06:54:11,036] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:11,593] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:11,632] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 06:54:11,900] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.188 [2020-06-04 06:54:12,440] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:12,942] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:13,033] - [remote_util] [139642271692544] - INFO - 172.23.105.188 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:54:13,175] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:54:13,176] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.188: systemctl stop couchbase-server.service [2020-06-04 06:54:15,109] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:15,110] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:54:15,225] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.188: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:54:15,247] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:15,247] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.188: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:54:15,331] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:15,335] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 06:54:15,604] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.188 [2020-06-04 06:54:16,157] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:16,641] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:16,729] - [remote_util] [139642271692544] - INFO - 172.23.105.188 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:54:16,729] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:54:16,867] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:54:16,998] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:54:17,000] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.188: systemctl start couchbase-server.service [2020-06-04 06:54:17,072] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:17,072] - [remote_util] [139642271692544] - INFO - 172.23.105.188:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:54:22,077] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.188: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:54:22,119] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:22,119] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 3961 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:54:22,120] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:54:22,234] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:54:22,352] - [remote_util] [139642271692544] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 06:54:22,452] - [remote_util] [139642271692544] - INFO - Connected to 172.23.108.238 [2020-06-04 06:54:22,650] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:23,154] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:23,184] - [remote_util] [139642271692544] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 06:54:23,283] - [remote_util] [139642271692544] - INFO - Connected to 172.23.108.238 [2020-06-04 06:54:23,451] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:23,854] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:23,940] - [remote_util] [139642271692544] - INFO - 172.23.108.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:54:24,213] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:54:24,215] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.108.238: systemctl stop couchbase-server.service [2020-06-04 06:54:25,843] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:25,843] - [basetestcase] [139642271692544] - INFO - Couchbase stopped [2020-06-04 06:54:25,958] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.108.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 06:54:25,978] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:25,978] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.108.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 06:54:26,062] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:26,066] - [remote_util] [139642271692544] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 06:54:26,165] - [remote_util] [139642271692544] - INFO - Connected to 172.23.108.238 [2020-06-04 06:54:26,347] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:26,856] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:26,941] - [remote_util] [139642271692544] - INFO - 172.23.108.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 06:54:26,942] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:54:27,042] - [remote_util] [139642271692544] - INFO - Starting couchbase server [2020-06-04 06:54:27,300] - [remote_util] [139642271692544] - INFO - Running systemd command on this server [2020-06-04 06:54:27,300] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.108.238: systemctl start couchbase-server.service [2020-06-04 06:54:27,325] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:27,326] - [remote_util] [139642271692544] - INFO - 172.23.108.238:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 06:54:32,331] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.108.238: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 06:54:32,356] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:54:32,356] - [remote_util] [139642271692544] - INFO - Couchbase server status: [' Process: 30295 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 06:54:32,357] - [remote_util] [139642271692544] - INFO - Checking for process beam.smp on linux [2020-06-04 06:54:32,459] - [basetestcase] [139642271692544] - INFO - Couchbase started [2020-06-04 06:54:32,573] - [basetestcase] [139642271692544] - INFO - sleep for 10 secs. ... [2020-06-04 06:54:42,672] - [task] [139638838318848] - INFO - server: ip:172.23.105.10 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:42,684] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.10', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.10', 'uptime': u'83', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.105.10:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3644870656, 'memoryTotal': 4219441152, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:42,696] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.10:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:42,697] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2156 [2020-06-04 06:54:42,711] - [rest_client] [139638838318848] - INFO - /node/controller/setupServices params on 172.23.105.10:8091:services=kv&password=password&user=Administrator&hostname=172.23.105.10 [2020-06-04 06:54:42,724] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.10:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:42,776] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:54:42,911] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.10 [2020-06-04 06:54:43,218] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:43,714] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:43,741] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:43,794] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:43,805] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.10:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:43,816] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.10:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:43,817] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:43,882] - [task] [139638838318848] - INFO - server: ip:172.23.105.122 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:43,894] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.122', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.122', 'uptime': u'77', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.105.122:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3829977088, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:43,906] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.122:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:43,906] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2217 [2020-06-04 06:54:43,919] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.122:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:43,977] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 06:54:44,076] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.122 [2020-06-04 06:54:44,386] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:44,810] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:44,835] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.122: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:44,872] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:44,887] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.122:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:44,898] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.122:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:44,899] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:44,954] - [task] [139638838318848] - INFO - server: ip:172.23.105.130 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:44,975] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.130', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.130', 'uptime': u'67', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.130:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3684245504, 'memoryTotal': 4201844736, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:44,987] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.130:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:44,988] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2147 [2020-06-04 06:54:45,002] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.130:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:45,067] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 06:54:45,203] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.130 [2020-06-04 06:54:45,514] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:45,981] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:46,007] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.130: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:46,057] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:46,076] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.130:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:46,088] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.130:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:46,089] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:46,145] - [task] [139638838318848] - INFO - server: ip:172.23.105.137 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:46,162] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.137', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.137', 'uptime': u'58', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.105.137:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3719376896, 'memoryTotal': 4219441152, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:46,183] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.137:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:46,184] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2156 [2020-06-04 06:54:46,199] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.137:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:46,259] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 06:54:46,667] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.137 [2020-06-04 06:54:46,950] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:47,480] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:47,517] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.137: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:47,571] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:47,592] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.137:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:47,612] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.137:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:47,612] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:47,683] - [task] [139638838318848] - INFO - server: ip:172.23.105.163 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:47,695] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.163', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.163', 'uptime': u'47', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.105.163:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3846365184, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:47,707] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.163:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:47,708] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2217 [2020-06-04 06:54:47,723] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.163:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:47,783] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 06:54:47,883] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.163 [2020-06-04 06:54:48,188] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:48,685] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:48,712] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.163: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:48,747] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:48,758] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.163:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:48,779] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.163:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:48,780] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:48,834] - [task] [139638838318848] - INFO - server: ip:172.23.105.185 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:48,846] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.185', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.185', 'uptime': u'37', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.105.185:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3806449664, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:48,857] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.185:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:48,858] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2217 [2020-06-04 06:54:48,882] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.185:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:48,928] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 06:54:49,027] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.185 [2020-06-04 06:54:49,316] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:49,742] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:49,771] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.185: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:49,800] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:49,811] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.185:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:49,823] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.185:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:49,824] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:49,892] - [task] [139638838318848] - INFO - server: ip:172.23.105.188 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:49,909] - [task] [139638838318848] - INFO - {'ip': u'172.23.105.188', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.188', 'uptime': u'25', 'mcdMemoryReserved': 3245, 'storageTotalRam': 4056, 'hostname': u'172.23.105.188:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3741487104, 'memoryTotal': 4254031872, 'memoryQuota': 1768, 'mcdMemoryAllocated': 3245, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:49,923] - [rest_client] [139638838318848] - ERROR - GET http://172.23.105.188:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:49,924] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2174 [2020-06-04 06:54:49,945] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.105.188:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:50,041] - [remote_util] [139638838318848] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 06:54:50,309] - [remote_util] [139638838318848] - INFO - Connected to 172.23.105.188 [2020-06-04 06:54:50,854] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:51,330] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:51,362] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.105.188: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:51,407] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:51,420] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.188:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:51,433] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.105.188:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:51,434] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:51,509] - [task] [139638838318848] - INFO - server: ip:172.23.108.238 port:8091 ssh_username:root, nodes/self [2020-06-04 06:54:51,524] - [task] [139638838318848] - INFO - {'ip': u'172.23.108.238', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'18', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3778, 'hostname': u'172.23.108.238:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3369459712, 'memoryTotal': 3961597952, 'memoryQuota': 1601, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 06:54:51,536] - [rest_client] [139638838318848] - ERROR - GET http://172.23.108.238:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 06:54:51,537] - [rest_client] [139638838318848] - INFO - pools/default params : memoryQuota=2024 [2020-06-04 06:54:51,551] - [rest_client] [139638838318848] - INFO - settings/web params on 172.23.108.238:8091:username=Administrator&password=password&port=8091 [2020-06-04 06:54:51,612] - [remote_util] [139638838318848] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 06:54:51,712] - [remote_util] [139638838318848] - INFO - Connected to 172.23.108.238 [2020-06-04 06:54:51,892] - [remote_util] [139638838318848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:54:52,396] - [remote_util] [139638838318848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:54:52,424] - [remote_util] [139638838318848] - INFO - running command.raw on 172.23.108.238: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:54:52,466] - [remote_util] [139638838318848] - INFO - command executed successfully [2020-06-04 06:54:52,486] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.108.238:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:52,502] - [rest_client] [139638838318848] - INFO - /diag/eval status on 172.23.108.238:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 06:54:52,502] - [rest_client] [139638838318848] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:54:52,571] - [basetestcase] [139642271692544] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.10 **** [2020-06-04 06:54:52,631] - [rest_client] [139642271692544] - ERROR - DELETE http://172.23.105.10:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2020-06-04 06:54:52,633] - [internal_user] [139642271692544] - INFO - Exception while deleting user. Exception is -"User was not found." [2020-06-04 06:54:52,723] - [basetestcase] [139642271692544] - INFO - sleep for 5 secs. ... [2020-06-04 06:54:57,728] - [basetestcase] [139642271692544] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2020-06-04 06:54:57,767] - [basetestcase] [139642271692544] - INFO - done initializing cluster [2020-06-04 06:54:58,552] - [task] [139638838318848] - INFO - adding node 172.23.105.122:8091 to cluster [2020-06-04 06:54:58,553] - [rest_client] [139638838318848] - INFO - adding remote node @172.23.105.122:8091 to this cluster @172.23.105.10:8091 [2020-06-04 06:54:59,985] - [task] [139638838318848] - INFO - adding node 172.23.105.130:8091 to cluster [2020-06-04 06:54:59,985] - [rest_client] [139638838318848] - INFO - adding remote node @172.23.105.130:8091 to this cluster @172.23.105.10:8091 [2020-06-04 06:55:01,740] - [task] [139638838318848] - INFO - adding node 172.23.105.137:8091 to cluster [2020-06-04 06:55:01,741] - [rest_client] [139638838318848] - INFO - adding remote node @172.23.105.137:8091 to this cluster @172.23.105.10:8091 [2020-06-04 06:55:03,664] - [task] [139638838318848] - INFO - adding node 172.23.105.163:8091 to cluster [2020-06-04 06:55:03,664] - [rest_client] [139638838318848] - INFO - adding remote node @172.23.105.163:8091 to this cluster @172.23.105.10:8091 [2020-06-04 06:55:05,106] - [rest_client] [139638838318848] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.163,ns_1@172.23.105.10,ns_1@172.23.105.137,ns_1@172.23.105.122'} [2020-06-04 06:55:05,127] - [rest_client] [139638838318848] - INFO - rebalance operation started [2020-06-04 06:55:05,141] - [rest_client] [139638838318848] - INFO - rebalance percentage : 0.00 % [2020-06-04 06:55:05,142] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 0.00% [2020-06-04 06:55:15,183] - [task] [139638838318848] - INFO - Rebalance - status: none, progress: 100.00% [2020-06-04 06:55:15,199] - [task] [139638838318848] - INFO - rebalancing was completed with progress: 100% in 10.0716590881 sec [2020-06-04 06:55:15,204] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:55:15,304] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 06:55:15,621] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:55:16,129] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:55:16,165] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 06:55:16,208] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 06:55:16,209] - [basetestcase] [139642271692544] - INFO - Enabled diag/eval for non-local hosts from 172.23.105.10 [2020-06-04 06:55:16,209] - [basetestcase] [139642271692544] - INFO - ============== basetestcase setup was finished for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2020-06-04 06:55:16,273] - [basetestcase] [139642271692544] - INFO - ------- Cluster statistics ------- [2020-06-04 06:55:16,273] - [basetestcase] [139642271692544] - INFO - 172.23.105.130:8091 => {'swap_mem_used': 44040192, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3630477312, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} [2020-06-04 06:55:16,274] - [basetestcase] [139642271692544] - INFO - 172.23.105.163:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.5012531328320802, 'mem_free': 3765071872, 'swap_mem_total': 3758092288, 'mem_total': 4339109888, 'services': [u'index', u'n1ql']} [2020-06-04 06:55:16,274] - [basetestcase] [139642271692544] - INFO - 172.23.105.10:8091 => {'swap_mem_used': 38535168, 'cpu_utilization': 0.5025125628140703, 'mem_free': 3596754944, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'kv']} [2020-06-04 06:55:16,275] - [basetestcase] [139642271692544] - INFO - 172.23.105.137:8091 => {'swap_mem_used': 786432, 'cpu_utilization': 0.5012531328320802, 'mem_free': 3666399232, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'eventing']} [2020-06-04 06:55:16,276] - [basetestcase] [139642271692544] - INFO - 172.23.105.122:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2506265664160401, 'mem_free': 3784835072, 'swap_mem_total': 3758092288, 'mem_total': 4339109888, 'services': [u'kv']} [2020-06-04 06:55:16,277] - [basetestcase] [139642271692544] - INFO - --- End of cluster statistics --- [2020-06-04 06:55:16,281] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 06:55:16,380] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 06:55:16,692] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 06:55:17,127] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 06:55:22,067] - [basetestcase] [139642271692544] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.163 port:8091 ssh_username:root] [2020-06-04 06:55:22,106] - [basetestcase] [139642271692544] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 06:55:22,123] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 06:55:22,142] - [eventing_base] [139642271692544] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2020-06-04 06:55:22,143] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=330 [2020-06-04 06:55:22,161] - [rest_client] [139642271692544] - INFO - pools/default params : indexMemoryQuota=256 [2020-06-04 06:55:22,189] - [rest_client] [139642271692544] - INFO - pools/default params : eventingMemoryQuota=512 [2020-06-04 06:55:22,234] - [basetestcase] [139642271692544] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.163 port:8091 ssh_username:root] [2020-06-04 06:55:22,235] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=700 [2020-06-04 06:55:22,248] - [eventing_rebalance] [139642271692544] - INFO - 100 [2020-06-04 06:55:23,324] - [rest_client] [139638838318848] - INFO - http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=100 [2020-06-04 06:55:23,341] - [rest_client] [139638838318848] - INFO - 0.02 seconds to create bucket src_bucket [2020-06-04 06:55:23,342] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:23,653] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:24,060] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:24,235] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:24,475] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:24,894] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:25,068] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:25,201] - [task] [139638838318848] - WARNING - vbucket map not ready after try 0 [2020-06-04 06:55:25,202] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:25,434] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:25,614] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:26,027] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:26,274] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:26,432] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:26,609] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:26,752] - [task] [139638838318848] - WARNING - vbucket map not ready after try 1 [2020-06-04 06:55:26,757] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:27,209] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:27,376] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:27,541] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:27,764] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:28,172] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:28,343] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:28,472] - [task] [139638838318848] - WARNING - vbucket map not ready after try 2 [2020-06-04 06:55:28,472] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:28,697] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:29,095] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:29,265] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:29,499] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:29,676] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:29,854] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:30,234] - [task] [139638838318848] - WARNING - vbucket map not ready after try 3 [2020-06-04 06:55:30,235] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:30,461] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:30,643] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:30,801] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:31,288] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:31,473] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:31,651] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:31,786] - [task] [139638838318848] - WARNING - vbucket map not ready after try 4 [2020-06-04 06:55:31,787] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:32,260] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:32,432] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:32,595] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:32,824] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:55:33,246] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:55:33,407] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:55:33,534] - [task] [139638838318848] - WARNING - vbucket map not ready after try 5 [2020-06-04 06:55:34,615] - [rest_client] [139638838318848] - INFO - http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=100 [2020-06-04 06:55:34,631] - [rest_client] [139638838318848] - INFO - 0.02 seconds to create bucket dst_bucket [2020-06-04 06:55:34,631] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:34,907] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:35,347] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:35,508] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:35,755] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:35,922] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:36,110] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:36,485] - [task] [139638838318848] - WARNING - vbucket map not ready after try 0 [2020-06-04 06:55:36,486] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:36,723] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:36,895] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:37,051] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:37,539] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:37,713] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:37,890] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:38,022] - [task] [139638838318848] - WARNING - vbucket map not ready after try 1 [2020-06-04 06:55:38,023] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:38,515] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:38,694] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:38,853] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:39,086] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:39,478] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:39,644] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:39,772] - [task] [139638838318848] - WARNING - vbucket map not ready after try 2 [2020-06-04 06:55:39,772] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:40,019] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:40,209] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:40,620] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:40,848] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:41,019] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:41,193] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:41,561] - [task] [139638838318848] - WARNING - vbucket map not ready after try 3 [2020-06-04 06:55:41,562] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:41,785] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:41,963] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:42,138] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:42,365] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:42,772] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:42,953] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:43,084] - [task] [139638838318848] - WARNING - vbucket map not ready after try 4 [2020-06-04 06:55:43,085] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 06:55:43,318] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:43,759] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:43,925] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:44,165] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 06:55:44,330] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 06:55:44,501] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 06:55:44,886] - [task] [139638838318848] - WARNING - vbucket map not ready after try 5 [2020-06-04 06:55:45,956] - [rest_client] [139638838318848] - INFO - http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=400 [2020-06-04 06:55:45,984] - [rest_client] [139638838318848] - INFO - 0.03 seconds to create bucket metadata [2020-06-04 06:55:45,985] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 06:55:46,294] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:46,455] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:46,637] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:47,121] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:47,295] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:47,456] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:47,586] - [task] [139638838318848] - WARNING - vbucket map not ready after try 0 [2020-06-04 06:55:47,591] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 06:55:47,823] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:48,238] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:48,430] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:48,698] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:48,868] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:49,292] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:49,415] - [task] [139638838318848] - WARNING - vbucket map not ready after try 1 [2020-06-04 06:55:49,415] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 06:55:49,637] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:49,810] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:49,990] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:50,475] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:50,636] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:50,821] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:50,943] - [task] [139638838318848] - WARNING - vbucket map not ready after try 2 [2020-06-04 06:55:50,944] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 06:55:51,422] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:51,597] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:51,763] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:52,011] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:52,434] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:52,604] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:52,725] - [task] [139638838318848] - WARNING - vbucket map not ready after try 3 [2020-06-04 06:55:52,725] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 06:55:52,947] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:53,362] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:53,543] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:53,783] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:53,951] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:54,126] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:54,507] - [task] [139638838318848] - WARNING - vbucket map not ready after try 4 [2020-06-04 06:55:54,507] - [bucket_helper] [139638838318848] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 06:55:54,757] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:54,951] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:55,125] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:55,637] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 06:55:55,811] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 06:55:55,981] - [data_helper] [139638838318848] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 06:55:56,113] - [task] [139638838318848] - WARNING - vbucket map not ready after try 5 [2020-06-04 06:55:58,575] - [eventing_base] [139642271692544] - INFO - Deploying the following handler code : Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes with bindings: {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [{'auth_type': 'no-auth', 'username': None, 'hostname': 'https://postman-echo.com/', 'validate_ssl_certificate': False, 'allow_cookies': False, 'value': 'server', 'password': None}], 'metadata_bucket': 'metadata'} and settings: {'cleanup_timers': False, 'worker_count': 3, 'skip_timer_threshold': 86400, 'dcp_stream_boundary': 'everything', 'use_memory_manager': True, 'sock_batch_size': 1, 'language_compatibility': '6.5.0', 'log_level': 'INFO', 'tick_duration': 5000, 'deadline_timeout': 21, 'timer_worker_pool_size': 3, 'deployment_status': True, 'description': 'Sample Description', 'dcp_gen_chan_size': 10000, 'timer_storage_chan_size': 10000, 'worker_queue_cap': 100000, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'processing_status': True, 'timer_processing_tick_interval': 500, 'data_chan_size': 10000, 'checkpoint_interval': 20000} [2020-06-04 06:55:58,575] - [eventing_base] [139642271692544] - INFO - function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { var request = { path : 'headers', headers: { "sample-header": "test" } }; try { var response = curl("GET", server, request); log('response body received from server:', response.body); log('response headers received from server:', response.headers); log('response status received from server:', response.status); var res= new Uint8Array(response.body); if(response.status == 200){ delete dst_bucket[context.docID]; } else{ delete dst_bucket[context.docID]; } } catch (e) { log('error:', e); } } function timerCallback(context) { var request = { path : 'headers', headers: { "sample-header": "test" } }; try { var response = curl("GET", server, request); log('response body received from server:', response.body); log('response headers received from server:', response.headers); log('response status received from server:', response.status); var res= new Uint8Array(response.body); if(response.status == 200){ dst_bucket[context.docID]=response.body; } else{ dst_bucket[context.docID]=response.status; } } catch (e) { log('error:', e); } } [2020-06-04 06:55:59,681] - [eventing_base] [139642271692544] - INFO - deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes' in metakv", "warnings": null } } [2020-06-04 06:55:59,683] - [basetestcase] [139642271692544] - INFO - sleep for 20 secs. Waiting for Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... [2020-06-04 06:56:19,726] - [basetestcase] [139642271692544] - INFO - sleep for 20 secs. Waiting for Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... [2020-06-04 06:56:39,919] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 06:56:40,109] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 06:56:40,277] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 06:56:41,336] - [task] [139638838318848] - INFO - adding node 172.23.105.185:8091 to cluster [2020-06-04 06:56:41,338] - [rest_client] [139638838318848] - INFO - adding remote node @172.23.105.185:8091 to this cluster @172.23.105.10:8091 [2020-06-04 06:56:42,953] - [task] [139638838318848] - INFO - adding node 172.23.105.188:8091 to cluster [2020-06-04 06:56:42,955] - [rest_client] [139638838318848] - INFO - adding remote node @172.23.105.188:8091 to this cluster @172.23.105.10:8091 [2020-06-04 06:56:43,561] - [rest_client] [139642271692544] - INFO - rebalance reached >100% in 3.01697087288 seconds [2020-06-04 06:56:46,900] - [rest_client] [139638838318848] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.188,ns_1@172.23.105.10,ns_1@172.23.105.137,ns_1@172.23.105.122,ns_1@172.23.105.163,ns_1@172.23.105.185'} [2020-06-04 06:56:46,936] - [rest_client] [139638838318848] - INFO - rebalance operation started [2020-06-04 06:56:46,955] - [rest_client] [139638838318848] - INFO - rebalance percentage : 0.00 % [2020-06-04 06:56:46,957] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 0.00% [2020-06-04 06:56:57,011] - [rest_client] [139638838318848] - INFO - rebalance percentage : 14.89 % [2020-06-04 06:56:57,012] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 14.89% [2020-06-04 06:57:07,059] - [rest_client] [139638838318848] - INFO - rebalance percentage : 23.69 % [2020-06-04 06:57:07,060] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 23.69% [2020-06-04 06:57:17,120] - [rest_client] [139638838318848] - INFO - rebalance percentage : 40.38 % [2020-06-04 06:57:17,121] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 40.38% [2020-06-04 06:57:27,178] - [rest_client] [139638838318848] - INFO - rebalance percentage : 48.03 % [2020-06-04 06:57:27,178] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 48.03% [2020-06-04 06:57:37,227] - [rest_client] [139638838318848] - INFO - rebalance percentage : 64.95 % [2020-06-04 06:57:37,227] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 64.95% [2020-06-04 06:57:47,286] - [rest_client] [139638838318848] - INFO - rebalance percentage : 78.57 % [2020-06-04 06:57:47,286] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 78.57% [2020-06-04 06:57:57,334] - [rest_client] [139638838318848] - INFO - rebalance percentage : 78.57 % [2020-06-04 06:57:57,335] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 78.57% [2020-06-04 06:58:07,381] - [rest_client] [139638838318848] - INFO - rebalance percentage : 82.58 % [2020-06-04 06:58:07,381] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 82.58% [2020-06-04 06:58:17,438] - [rest_client] [139638838318848] - INFO - rebalance percentage : 88.07 % [2020-06-04 06:58:17,439] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 88.07% [2020-06-04 06:58:27,486] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.09 % [2020-06-04 06:58:27,487] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.09% [2020-06-04 06:58:37,534] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.16 % [2020-06-04 06:58:37,534] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.16% [2020-06-04 06:58:47,581] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.30 % [2020-06-04 06:58:47,582] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.30% [2020-06-04 06:58:57,639] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.30 % [2020-06-04 06:58:57,641] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.30% [2020-06-04 06:59:07,692] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.37 % [2020-06-04 06:59:07,693] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.37% [2020-06-04 06:59:17,739] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.50 % [2020-06-04 06:59:17,739] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.50% [2020-06-04 06:59:27,801] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.55 % [2020-06-04 06:59:27,802] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.55% [2020-06-04 06:59:37,849] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.66 % [2020-06-04 06:59:37,850] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.66% [2020-06-04 06:59:47,898] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.73 % [2020-06-04 06:59:47,898] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.73% [2020-06-04 06:59:57,944] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.83 % [2020-06-04 06:59:57,945] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.83% [2020-06-04 07:00:08,007] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.89 % [2020-06-04 07:00:08,009] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.89% [2020-06-04 07:00:18,056] - [rest_client] [139638838318848] - INFO - rebalance percentage : 89.99 % [2020-06-04 07:00:18,056] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 89.99% [2020-06-04 07:00:28,104] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.10 % [2020-06-04 07:00:28,105] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.10% [2020-06-04 07:00:38,171] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.22 % [2020-06-04 07:00:38,172] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.22% [2020-06-04 07:00:48,219] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.31 % [2020-06-04 07:00:48,220] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.31% [2020-06-04 07:00:58,270] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.36 % [2020-06-04 07:00:58,271] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.36% [2020-06-04 07:01:08,334] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.43 % [2020-06-04 07:01:08,336] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.43% [2020-06-04 07:01:18,384] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.45 % [2020-06-04 07:01:18,384] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.45% [2020-06-04 07:01:28,432] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.47 % [2020-06-04 07:01:28,432] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.47% [2020-06-04 07:01:38,486] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.56 % [2020-06-04 07:01:38,487] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.56% [2020-06-04 07:01:48,541] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.65 % [2020-06-04 07:01:48,541] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.65% [2020-06-04 07:01:58,588] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.72 % [2020-06-04 07:01:58,589] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.72% [2020-06-04 07:02:08,636] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.77 % [2020-06-04 07:02:08,636] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.77% [2020-06-04 07:02:18,700] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.82 % [2020-06-04 07:02:18,701] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.82% [2020-06-04 07:02:28,749] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.86 % [2020-06-04 07:02:28,750] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.86% [2020-06-04 07:02:38,800] - [rest_client] [139638838318848] - INFO - rebalance percentage : 90.97 % [2020-06-04 07:02:38,800] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 90.97% [2020-06-04 07:02:48,848] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.02 % [2020-06-04 07:02:48,849] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.02% [2020-06-04 07:02:58,902] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.05 % [2020-06-04 07:02:58,902] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.05% [2020-06-04 07:03:08,948] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.12 % [2020-06-04 07:03:08,949] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.12% [2020-06-04 07:03:18,996] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.12 % [2020-06-04 07:03:18,996] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.12% [2020-06-04 07:03:29,057] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.20 % [2020-06-04 07:03:29,062] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.20% [2020-06-04 07:03:39,120] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.32 % [2020-06-04 07:03:39,120] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.32% [2020-06-04 07:03:49,169] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.43 % [2020-06-04 07:03:49,169] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.43% [2020-06-04 07:03:59,216] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.48 % [2020-06-04 07:03:59,216] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.48% [2020-06-04 07:04:09,269] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.55 % [2020-06-04 07:04:09,269] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.55% [2020-06-04 07:04:19,318] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.60 % [2020-06-04 07:04:19,318] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.60% [2020-06-04 07:04:29,366] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.65 % [2020-06-04 07:04:29,367] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.65% [2020-06-04 07:04:39,430] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.73 % [2020-06-04 07:04:39,431] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.73% [2020-06-04 07:04:49,480] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.78 % [2020-06-04 07:04:49,480] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.78% [2020-06-04 07:04:59,527] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.85 % [2020-06-04 07:04:59,527] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.85% [2020-06-04 07:05:09,574] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.88 % [2020-06-04 07:05:09,575] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.88% [2020-06-04 07:05:19,634] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.96 % [2020-06-04 07:05:19,635] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.96% [2020-06-04 07:05:29,684] - [rest_client] [139638838318848] - INFO - rebalance percentage : 91.99 % [2020-06-04 07:05:29,684] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 91.99% [2020-06-04 07:05:39,733] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.04 % [2020-06-04 07:05:39,734] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.04% [2020-06-04 07:05:49,803] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.08 % [2020-06-04 07:05:49,804] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.08% [2020-06-04 07:05:59,853] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.17 % [2020-06-04 07:05:59,853] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.17% [2020-06-04 07:06:09,900] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.22 % [2020-06-04 07:06:09,901] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.22% [2020-06-04 07:06:19,948] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.29 % [2020-06-04 07:06:19,949] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.29% [2020-06-04 07:06:29,998] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.36 % [2020-06-04 07:06:29,998] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.36% [2020-06-04 07:06:40,047] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.40 % [2020-06-04 07:06:40,048] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.40% [2020-06-04 07:06:50,094] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.50 % [2020-06-04 07:06:50,095] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.50% [2020-06-04 07:07:00,140] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.63 % [2020-06-04 07:07:00,142] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.63% [2020-06-04 07:07:10,190] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.66 % [2020-06-04 07:07:10,191] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.66% [2020-06-04 07:07:20,240] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.73 % [2020-06-04 07:07:20,240] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.73% [2020-06-04 07:07:30,287] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.80 % [2020-06-04 07:07:30,288] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.80% [2020-06-04 07:07:40,350] - [rest_client] [139638838318848] - INFO - rebalance percentage : 92.84 % [2020-06-04 07:07:40,351] - [task] [139638838318848] - INFO - Rebalance - status: running, progress: 92.84% [2020-06-04 07:07:50,398] - [task] [139638838318848] - INFO - Rebalance - status: none, progress: 100.00% [2020-06-04 07:07:50,420] - [task] [139638838318848] - INFO - rebalancing was completed with progress: 100% in 663.483546972 sec [2020-06-04 07:07:50,448] - [basetestcase] [139642271692544] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:07:50,507] - [basetestcase] [139642271692544] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:07:50,836] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 6573 Expected : 10080 ... [2020-06-04 07:08:21,356] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 6915 Expected : 10080 ... [2020-06-04 07:08:51,722] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 7254 Expected : 10080 ... [2020-06-04 07:09:21,911] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 7642 Expected : 10080 ... [2020-06-04 07:09:52,095] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 7995 Expected : 10080 ... [2020-06-04 07:10:22,304] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 8410 Expected : 10080 ... [2020-06-04 07:10:52,521] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 8704 Expected : 10080 ... [2020-06-04 07:11:22,772] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9058 Expected : 10080 ... [2020-06-04 07:11:53,058] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9416 Expected : 10080 ... [2020-06-04 07:12:23,311] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9735 Expected : 10080 ... [2020-06-04 07:12:53,839] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9940 Expected : 10080 ... [2020-06-04 07:13:24,046] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10054 Expected : 10080 ... [2020-06-04 07:13:54,380] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:14:24,762] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:14:54,991] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:15:25,172] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:15:56,031] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:16:26,323] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:16:56,717] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:17:27,037] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:17:57,265] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:18:27,487] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:18:57,814] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:19:27,998] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:19:58,293] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:20:28,533] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:20:58,774] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:21:29,035] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:21:59,282] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:22:29,481] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:22:59,743] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:23:29,967] - [basetestcase] [139642271692544] - INFO - sleep for 30 secs. Waiting for handler code Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10078 Expected : 10080 ... [2020-06-04 07:24:01,000] - [eventing_base] [139642271692544] - INFO - Documents in source bucket : 10080 [2020-06-04 07:24:09,637] - [eventing_base] [139642271692544] - INFO - Stats for Node 172.23.105.137 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 77, "agg_messages_sent_to_worker": 46874, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 18695, "dcp_mutation_sent_to_worker": 18695, "dcp_snapshot": 11133, "dcp_stream_req_counter": 3000, "dcp_stream_req_err_counter": 693, "dcp_streamend": 1053, "dcp_streamreq": 2307, "execution_stats": 5005, "failure_stats": 5005, "latency_stats": 5005, "lcb_exception_stats": 5005, "log_level": 3, "num_processed_events": 18695, "processed_events_size": 12624944, "thr_count": 3, "thr_map": 3, "timer_events": 10080, "timer_responses_received": 10080, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 10080, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 18695, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 18695, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 8615, "lcb_retry_failure": 0, "messages_parsed": 46862, "num_processed_events": 18695, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 10080, "processed_events_size": 12624944, "timer_cancel_counter": 0, "timer_create_counter": 10080, "timer_create_failure": 0, "timer_msg_counter": 10080, "timer_responses_sent": 0, "timestamp": { "7210": "2020-06-04T14:24:00Z", "7222": "2020-06-04T14:24:00Z", "7233": "2020-06-04T14:24:00Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 1, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "7210": "2020-06-04T14:24:00Z", "7222": "2020-06-04T14:24:00Z", "7233": "2020-06-04T14:24:00Z" }, "v8worker_events_lost": 0 }, "function_id": 235592861, "function_name": "Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 24, "internal_vb_distribution_stats": { "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 671500, "50": 11700, "80": 16500, "90": 20900, "95": 29600, "99": 233900 }, "lcb_creds_request_counter": 66, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.137:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.137:8096": { "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_0": 7210, "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_1": 7233, "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_2": 7222 } } ] [2020-06-04 07:24:10,553] - [basetestcase] [139642271692544] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:24:10,589] - [eventing_base] [139642271692544] - INFO - Go routine dumps for Node 172.23.105.137 is goroutine profile: total 255 20 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c548 0xce6f50 0xce6ffa 0xcfaff8 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c547 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xce6f4f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0xce6ff9 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0xcfaff7 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf7 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:286 20 @ 0x913f1a 0x913fce 0x8eb992 0x8eb64b 0xcfbae9 0x9417b1 # 0xcfbae8 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 20 @ 0x913f1a 0x913fce 0x9259db 0x947c90 0xce8230 0xce7ae3 0xce9a0e 0xcea7a0 0x9417b1 # 0x9259da sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x947c8f sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0xce822f gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0xce7ae2 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0xce9a0d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0xcea79f gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 20 @ 0x913f1a 0x923c30 0xcfac72 0x9417b1 # 0xcfac71 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:257 14 @ 0x913f1a 0x923c30 0xb86b3b 0x9417b1 # 0xb86b3a net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 11 @ 0x913f1a 0x923c30 0xb85b33 0x9417b1 # 0xb85b32 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 9 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2cb9c 0xb2ed24 0xb2e802 0xb2eae3 0xb7c0d1 0xb7c01d 0xb8827c 0x9e5d52 0x9e5a5d 0x9e54a8 0xc4b91e 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb2ed23 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb2e801 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb2eae2 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb7c0d0 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb7c01c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb8827b net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e5d51 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e5a5c encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e54a7 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc4b91d github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:288 7 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0x969f36 0x96a0a8 0xc6f5a9 0xc7b6b1 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x969f35 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x96a0a7 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0xc6f5a8 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0xc7b6b0 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x220 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1116 7 @ 0x913f1a 0x913fce 0x8eb992 0x8eb64b 0xc8f95e 0x9417b1 # 0xc8f95d github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:307 7 @ 0x913f1a 0x923c30 0x10f4e7f 0x9417b1 # 0x10f4e7e github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15e /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:623 7 @ 0x913f1a 0x923c30 0xc73d9b 0x9417b1 # 0xc73d9a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:191 5 @ 0x913f1a 0x923c30 0xd72fe3 0x9417b1 # 0xd72fe2 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 4 @ 0x913f1a 0x923c30 0xccf43f 0x9417b1 # 0xccf43e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 4 @ 0x913f1a 0x923c30 0xd34da4 0x9417b1 # 0xd34da3 gopkg.in/couchbase/gocb%2ev1.(*ThresholdLoggingTracer).loggerRoutine+0x133 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/thresholdlogtracer.go:244 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0x9b344e 0x9b01ea 0xa2bebe 0xa2cb9c 0xa2cdb4 0x10effaa 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9b344d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9b01e9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa2cdb3 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10effa9 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:103 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0x9b344e 0x9b01ea 0xa2bebe 0xa2cb9c 0xa2cdb4 0x10f0252 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9b344d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9b01e9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa2cdb3 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10f0251 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:116 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c43c 0x10a4a85 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c43b bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x10a4a84 github.com/couchbase/eventing/consumer.(*Consumer).feedbackReadMessageLoop+0x2b4 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:749 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c43c 0x10a51aa 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c43b bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x10a51a9 github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop+0x2a9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:813 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2c00a 0xb85575 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2c009 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb85574 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9abd88 0xa74602 0xa954a2 0xa93469 0x11243d1 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9abd87 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa74601 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa954a1 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa93468 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x11243d0 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:604 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9abd88 0xa74602 0xa954a2 0xa93469 0x1124d51 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9abd87 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa74601 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa954a1 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa93468 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x1124d50 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:638 3 @ 0x913f1a 0x913fce 0x9248b4 0x9244d9 0x94ac42 0x10ceada 0xd755a3 0x9417b1 # 0x9244d8 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x94ac41 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0x10cead9 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1c19 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:295 # 0xd755a2 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 3 @ 0x913f1a 0x923c30 0x108704e 0x9417b1 # 0x108704d github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ad /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 3 @ 0x913f1a 0x923c30 0x108b214 0x9417b1 # 0x108b213 github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x1e3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x913f1a 0x923c30 0x10a2cde 0x9417b1 # 0x10a2cdd github.com/couchbase/eventing/consumer.(*Consumer).sendMessageLoop+0x18d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:582 3 @ 0x913f1a 0x923c30 0x10a85c9 0x9417b1 # 0x10a85c8 github.com/couchbase/eventing/consumer.(*Consumer).processDCPEvents+0x748 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:57 3 @ 0x913f1a 0x923c30 0x10ad3a8 0x9417b1 # 0x10ad3a7 github.com/couchbase/eventing/consumer.(*Consumer).processStatsEvents+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:329 3 @ 0x913f1a 0x923c30 0x10ae233 0x9417b1 # 0x10ae232 github.com/couchbase/eventing/consumer.(*Consumer).processFilterEvents+0x2e2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:379 3 @ 0x913f1a 0x923c30 0x10b8922 0x9417b1 # 0x10b8921 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x161 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:946 3 @ 0x913f1a 0x923c30 0x10bc7ce 0x9417b1 # 0x10bc7cd github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1168 3 @ 0x913f1a 0x923c30 0x10c8721 0x9417b1 # 0x10c8720 github.com/couchbase/eventing/consumer.(*Consumer).loadStatsFromConsumer+0x1a0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:105 3 @ 0x913f1a 0x923c30 0x10c9264 0x9417b1 # 0x10c9263 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0xf3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:141 3 @ 0x913f1a 0x923c30 0x11245a5 0x9417b1 # 0x11245a4 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:608 3 @ 0x913f1a 0x923c30 0x1124f25 0x9417b1 # 0x1124f24 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:642 3 @ 0x9558e5 0x9b6098 0x9afa7c 0x9af02b 0xd8292c 0x1089e82 0xd755a3 0x9417b1 # 0x9558e4 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x9b6097 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x9afa7b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x9af02a os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xd8292b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0x1089e81 github.com/couchbase/eventing/consumer.(*client).Serve+0x1171 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:126 # 0xd755a2 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 2 @ 0x913f1a 0x923c30 0x1126a1c 0x9417b1 # 0x1126a1b github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 2 @ 0x913f1a 0x923c30 0xd563a1 0x9417b1 # 0xd563a0 github.com/couchbase/eventing/util.(*Stats).updater+0x100 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/stats.go:54 1 @ 0x8f6ab2 0x9289c6 0xd97232 0x9417b1 # 0x9289c5 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd97231 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c548 0xc005dd 0x9e5d52 0x9e5a5d 0x9e54a8 0xbff15f 0xbfce27 0xbfc71a 0xbfbd87 0xc00b49 0xc013ac 0x11268ad 0x115b93e 0x1179290 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c547 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xc005dc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e5d51 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e5a5c encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e54a7 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbff15e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbfce26 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbfc719 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbfbd86 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xc00b48 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xc013ab github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x11268ac github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x115b93d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:444 # 0x117928f github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c548 0xc005dd 0x9e5d52 0x9e5a5d 0x9e54a8 0xbff15f 0xbfce27 0xbfc71a 0xbfbd87 0xc00b49 0xc013ac 0xc44880 0xc45f35 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c547 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xc005dc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e5d51 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e5a5c encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e54a7 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbff15e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbfce26 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbfc719 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbfbd86 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xc00b48 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xc013ab github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc4487f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc45f34 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb6d188 0xa2bebe 0xa2c00a 0xb72413 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb6d187 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2c009 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb72412 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2cb9c 0xb2ed24 0xb2e802 0xb2eae3 0xb7c0d1 0xb7c01d 0xb8827c 0xa2bebe 0xa2cb9c 0xa2d05b 0xc8a507 0xc89e4e 0xd53783 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb2ed23 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb2e801 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb2eae2 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb7c0d0 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb7c01c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb8827b net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa2d05a bufio.(*Reader).ReadBytes+0x6a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:419 # 0xc8a506 github.com/couchbase/eventing/dcp.(*Client).runObserveStreamingEndpoint+0x526 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:409 # 0xc89e4d github.com/couchbase/eventing/dcp.(*Client).RunObservePool+0xdd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:345 # 0xd53782 github.com/couchbase/eventing/util.(*serviceNotifierInstance).RunPoolObserver+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/services_notifier.go:79 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2cb9c 0xb2ed24 0xb2e802 0xb2eae3 0xb7c0d1 0xb7c01d 0xb8827c 0xa2bebe 0xa2cb9c 0xa2d05b 0xc8a507 0xc89f96 0xd53953 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb2ed23 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb2e801 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb2eae2 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb7c0d0 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb7c01c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb8827b net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa2d05a bufio.(*Reader).ReadBytes+0x6a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:419 # 0xc8a506 github.com/couchbase/eventing/dcp.(*Client).runObserveStreamingEndpoint+0x526 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:409 # 0xc89f95 github.com/couchbase/eventing/dcp.(*Client).RunObserveNodeServices+0xf5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:361 # 0xd53952 github.com/couchbase/eventing/util.(*serviceNotifierInstance).RunServicesObserver+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/services_notifier.go:88 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9abd88 0xa74602 0xa8e94e 0xa8cf29 0xb775cf 0xaf2407 0xb76365 0xb7697b 0xb7726e 0x11799f6 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9abd87 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa74601 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa8e94d net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa8cf28 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb775ce net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaf2406 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb76364 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb7697a net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb7726d net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x11799f5 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:227 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9abd88 0xa74602 0xa8e94e 0xa8cf29 0xb775cf 0xb76365 0xb760b9 0x1179477 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9abd87 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa74601 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa8e94d net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa8cf28 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb775ce net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb76364 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb760b8 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x1179476 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:185 1 @ 0x913f1a 0x913fce 0x8eb992 0x8eb64b 0x118ea55 0x119a14c 0x913ac2 0x9417b1 # 0x118ea54 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x119a14b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x913ac1 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x913f1a 0x913fce 0x8eb992 0x8eb64b 0x1194f53 0x9417b1 # 0x1194f52 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:84 1 @ 0x913f1a 0x913fce 0x8eb992 0x8eb68b 0xd509c7 0x9417b1 # 0xd509c6 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x913f1a 0x913fce 0x930476 0x110b924 0x9417b1 # 0x930475 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x110b923 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:141 1 @ 0x913f1a 0x913fce 0x930476 0xd52123 0x9417b1 # 0x930475 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xd52122 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x913f1a 0x923c30 0x110e7d0 0xd755a3 0x9417b1 # 0x110e7cf github.com/couchbase/eventing/producer.(*Producer).Serve+0x185f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:202 # 0xd755a2 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x913f1a 0x923c30 0x1116eb5 0x9417b1 # 0x1116eb4 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:903 1 @ 0x913f1a 0x923c30 0x115c537 0x11612c2 0x1126ce4 0x112610b 0x1126503 0x11276a2 0x93f06b 0x9911d9 0x990754 0xbfb4ae 0x9417b1 # 0x115c536 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:541 # 0x11612c1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1126ce3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x112610a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1126502 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93f06a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x9911d8 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x990753 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbfb4ad net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x913f1a 0x923c30 0x115c537 0x1161a52 0x1126b34 0x112610b 0x11263a3 0x11275d2 0x93f06b 0x9911d9 0x990754 0xbfb4ae 0x9417b1 # 0x115c536 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:541 # 0x1161a51 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1126b33 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x112610a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11263a2 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93f06a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x9911d8 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x990753 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbfb4ad net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x913f1a 0x923c30 0x1192f5a 0x9417b1 # 0x1192f59 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).watchBucketChanges+0x409 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/util.go:244 1 @ 0x913f1a 0x923c30 0xb9e321 0x9417b1 # 0xb9e320 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x913f1a 0x923c30 0xb9e78f 0x9417b1 # 0xb9e78e github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x1179d2c 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x1179d2b github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0xab /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:241 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x1179ecc 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x1179ecb github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0xab /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:252 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x117a06c 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x117a06b github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0xab /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:263 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119a896 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119a895 main.main.func1+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119a9d6 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119a9d5 main.main.func2+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119ab16 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119ab15 main.main.func3+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119ac56 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119ac55 main.main.func4+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119ad96 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119ad95 main.main.func5+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119aed6 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119aed5 main.main.func6+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0x913f1a 0x923c30 0xd08263 0xd08a0f 0xd06ed6 0x111f8eb 0xd530ba 0x1101f93 0x1116a4a 0x9417b1 # 0xd08262 gopkg.in/couchbase/gocb%2ev1.(*opManager).Wait+0x112 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/bucket_crud.go:163 # 0xd08a0e gopkg.in/couchbase/gocb%2ev1.(*Bucket).get+0x23e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/bucket_crud.go:210 # 0xd06ed5 gopkg.in/couchbase/gocb%2ev1.(*Bucket).Get+0xe5 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/bucket_crud.go:13 # 0x111f8ea github.com/couchbase/eventing/producer.glob..func9+0x14a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/bucket_ops.go:257 # 0xd530b9 github.com/couchbase/eventing/util.Retry+0x139 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry.go:65 # 0x1101f92 github.com/couchbase/eventing/producer.(*Producer).vbDistributionStats+0x362 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/exported_functions.go:417 # 0x1116a49 github.com/couchbase/eventing/producer.(*Producer).updateStats+0x3f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:854 1 @ 0x913f1a 0x923c30 0xd50039 0x9417b1 # 0xd50038 github.com/couchbase/eventing/util.(*ClusterInfoClient).watchClusterChanges+0x328 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:729 1 @ 0x955865 0x9532af 0x951739 0x9aa788 0xa73cef 0xa84fca 0xb6ccda 0x9417b1 # 0x955864 syscall.Syscall+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:18 # 0x9532ae syscall.read+0x5e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/zsyscall_linux_amd64.go:749 # 0x951738 syscall.Read+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/syscall_unix.go:162 # 0x9aa787 internal/poll.(*FD).Read+0x117 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:153 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb6ccd9 net/http.(*connReader).backgroundRead+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:668 1 @ 0xd8e858 0xd8e660 0xd8b1a4 0xd9696d 0xd96cf1 0x11559ed 0x117ad88 0xb73284 0xb74ef0 0xb75f9c 0xb722a1 0x9417b1 # 0xd8e857 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd8e65f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd8b1a3 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd9696c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd96cf0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11559ec github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3571 # 0x117ad87 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb73283 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb74eef net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb75f9b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb722a0 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2020-06-04 07:24:10,622] - [basetestcase] [139642271692544] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:24:10,671] - [eventing_base] [139642271692544] - INFO - Stats for Node 172.23.105.137 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 77, "agg_messages_sent_to_worker": 47009, "agg_queue_memory": 4, "agg_queue_memory_cap": 106954752, "agg_queue_size": 1, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 18695, "dcp_mutation_sent_to_worker": 18695, "dcp_snapshot": 11133, "dcp_stream_req_counter": 3000, "dcp_stream_req_err_counter": 693, "dcp_streamend": 1053, "dcp_streamreq": 2307, "execution_stats": 5032, "failure_stats": 5032, "latency_stats": 5032, "lcb_exception_stats": 5032, "log_level": 3, "num_processed_events": 18695, "processed_events_size": 12624944, "thr_count": 3, "thr_map": 3, "timer_events": 10080, "timer_responses_received": 10080, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 4, "agg_queue_size": 1, "curl": { "delete": 0, "get": 10080, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 18695, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 18695, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 8615, "lcb_retry_failure": 0, "messages_parsed": 46997, "num_processed_events": 18695, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 10080, "processed_events_size": 12624944, "timer_cancel_counter": 0, "timer_create_counter": 10080, "timer_create_failure": 0, "timer_msg_counter": 10080, "timer_responses_sent": 0, "timestamp": { "7210": "2020-06-04T14:24:09Z", "7222": "2020-06-04T14:24:09Z", "7233": "2020-06-04T14:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 1, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "7210": "2020-06-04T14:24:09Z", "7222": "2020-06-04T14:24:09Z", "7233": "2020-06-04T14:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 235592861, "function_name": "Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 24, "internal_vb_distribution_stats": { "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 671500, "50": 11700, "80": 16500, "90": 20900, "95": 29600, "99": 233900 }, "lcb_creds_request_counter": 66, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.137:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.137:8096": { "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_0": 7210, "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_1": 7233, "worker_Function_927244291_test_eventing_rebalance_with_multiple_kv_nodes_2": 7222 } } ] [2020-06-04 07:24:10,725] - [basetestcase] [139642271692544] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:24:10,748] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:24:10,889] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.137 [2020-06-04 07:24:11,194] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:24:11,685] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:24:11,714] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.137:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-06-04 07:24:11,714] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-06-04 07:24:12,142] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:24:12,144] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-06-04 07:24:12,197] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:24:12,925] - [eventing_base] [139642271692544] - INFO - Bucket dst_bucket DGM is 100 [2020-06-04 07:24:13,016] - [eventing_base] [139642271692544] - INFO - Bucket metadata DGM is 100 [2020-06-04 07:24:13,089] - [eventing_base] [139642271692544] - INFO - Bucket src_bucket DGM is 100 [2020-06-04 07:24:13,160] - [eventing_base] [139642271692544] - INFO - number of documents in metadata bucket 2048 [2020-06-04 07:24:13,282] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:24:13,414] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:24:13,691] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:24:14,123] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:24:14,124] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: /opt/couchbase/bin/cbcollect_info 172.23.105.10-20200604-0724-diag.zip [2020-06-04 07:26:33,573] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:26:36,845] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: rm -f /root/172.23.105.10-20200604-0724-diag.zip [2020-06-04 07:26:36,946] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:26:37,071] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:26:37,171] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:26:37,450] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:26:37,937] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:26:37,938] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: /opt/couchbase/bin/cbcollect_info 172.23.105.122-20200604-0726-diag.zip [2020-06-04 07:28:14,762] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:28:16,371] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: rm -f /root/172.23.105.122-20200604-0726-diag.zip [2020-06-04 07:28:16,455] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:28:16,573] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:28:16,709] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:28:17,037] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:28:17,474] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:28:17,475] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: /opt/couchbase/bin/cbcollect_info 172.23.105.130-20200604-0728-diag.zip [2020-06-04 07:30:11,345] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:30:12,952] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: rm -f /root/172.23.105.130-20200604-0728-diag.zip [2020-06-04 07:30:13,039] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:30:13,046] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:30:13,218] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:30:13,617] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:30:14,126] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:30:14,127] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: /opt/couchbase/bin/cbcollect_info 172.23.105.137-20200604-0730-diag.zip [2020-06-04 07:31:45,093] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:31:46,328] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: rm -f /root/172.23.105.137-20200604-0730-diag.zip [2020-06-04 07:31:46,367] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:31:46,492] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:31:46,592] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:31:46,891] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:31:47,346] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:31:47,347] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: /opt/couchbase/bin/cbcollect_info 172.23.105.163-20200604-0731-diag.zip [2020-06-04 07:33:35,237] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:33:36,380] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: rm -f /root/172.23.105.163-20200604-0731-diag.zip [2020-06-04 07:33:36,464] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:33:36,582] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:33:36,681] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:33:36,971] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:33:37,417] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:33:37,418] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: /opt/couchbase/bin/cbcollect_info 172.23.105.185-20200604-0733-diag.zip [2020-06-04 07:35:16,060] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:35:17,583] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: rm -f /root/172.23.105.185-20200604-0733-diag.zip [2020-06-04 07:35:17,667] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:35:17,788] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:35:18,056] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:35:18,581] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:35:18,980] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:35:18,982] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: /opt/couchbase/bin/cbcollect_info 172.23.105.188-20200604-0735-diag.zip [2020-06-04 07:38:11,385] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:38:13,099] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: rm -f /root/172.23.105.188-20200604-0735-diag.zip [2020-06-04 07:38:13,184] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:38:13,195] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:38:13,295] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:38:13,464] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:38:13,975] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:38:13,978] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: /opt/couchbase/bin/cbcollect_info 172.23.108.238-20200604-0738-diag.zip [2020-06-04 07:39:30,576] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:39:31,033] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: rm -f /root/172.23.108.238-20200604-0738-diag.zip [2020-06-04 07:39:31,118] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:39:31,272] - [eventing_base] [139645367297856] - INFO - Starting Test: suite_tearDown [2020-06-04 07:39:31,277] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:39:31,412] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:39:31,707] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:39:32,202] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:39:32,323] - [rest_client] [139645367297856] - INFO - Node version in cluster 6.6.0-7737-enterprise [2020-06-04 07:39:32,366] - [rest_client] [139645367297856] - INFO - Node versions in cluster [u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise', u'6.6.0-7737-enterprise'] [2020-06-04 07:39:32,369] - [basetestcase] [139645367297856] - INFO - ============== basetestcase setup was started for test #8 suite_tearDown============== [2020-06-04 07:39:32,510] - [bucket_helper] [139645367297856] - INFO - deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket'] on 172.23.105.10 [2020-06-04 07:39:32,510] - [bucket_helper] [139645367297856] - INFO - remove bucket dst_bucket ... [2020-06-04 07:39:33,115] - [bucket_helper] [139645367297856] - INFO - deleted bucket : dst_bucket from 172.23.105.10 [2020-06-04 07:39:33,116] - [bucket_helper] [139645367297856] - INFO - waiting for bucket deletion to complete.... [2020-06-04 07:39:33,189] - [rest_client] [139645367297856] - INFO - node 172.23.105.10 existing buckets : [u'metadata', u'src_bucket'] [2020-06-04 07:39:33,193] - [bucket_helper] [139645367297856] - INFO - remove bucket metadata ... [2020-06-04 07:39:33,726] - [bucket_helper] [139645367297856] - INFO - deleted bucket : metadata from 172.23.105.10 [2020-06-04 07:39:33,727] - [bucket_helper] [139645367297856] - INFO - waiting for bucket deletion to complete.... [2020-06-04 07:39:33,761] - [rest_client] [139645367297856] - INFO - node 172.23.105.10 existing buckets : [u'src_bucket'] [2020-06-04 07:39:33,765] - [bucket_helper] [139645367297856] - INFO - remove bucket src_bucket ... [2020-06-04 07:39:35,099] - [bucket_helper] [139645367297856] - INFO - deleted bucket : src_bucket from 172.23.105.10 [2020-06-04 07:39:35,100] - [bucket_helper] [139645367297856] - INFO - waiting for bucket deletion to complete.... [2020-06-04 07:39:35,112] - [rest_client] [139645367297856] - INFO - node 172.23.105.10 existing buckets : [] [2020-06-04 07:39:35,112] - [bucket_helper] [139645367297856] - INFO - sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket']) were deleted completely. [2020-06-04 07:39:37,381] - [cluster_helper] [139645367297856] - INFO - rebalancing all nodes in order to remove nodes [2020-06-04 07:39:37,399] - [rest_client] [139645367297856] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.188,ns_1@172.23.105.137,ns_1@172.23.105.122,ns_1@172.23.105.163,ns_1@172.23.105.185', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.188,ns_1@172.23.105.10,ns_1@172.23.105.137,ns_1@172.23.105.122,ns_1@172.23.105.163,ns_1@172.23.105.185'} [2020-06-04 07:39:37,412] - [rest_client] [139645367297856] - INFO - rebalance operation started [2020-06-04 07:39:37,427] - [rest_client] [139645367297856] - INFO - rebalance percentage : 0.00 % [2020-06-04 07:39:47,449] - [rest_client] [139645367297856] - INFO - rebalance percentage : 78.57 % [2020-06-04 07:39:57,472] - [rest_client] [139645367297856] - INFO - rebalance percentage : 92.86 % [2020-06-04 07:40:17,510] - [rest_client] [139645367297856] - INFO - rebalance progress took 40.10 seconds [2020-06-04 07:40:17,511] - [rest_client] [139645367297856] - INFO - sleep for 10 seconds after rebalance... [2020-06-04 07:40:27,684] - [cluster_helper] [139645367297856] - INFO - removed all the nodes from cluster associated with ip:172.23.105.10 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.130', 8091), (u'ns_1@172.23.105.188', 8091), (u'ns_1@172.23.105.137', 8091), (u'ns_1@172.23.105.122', 8091), (u'ns_1@172.23.105.163', 8091), (u'ns_1@172.23.105.185', 8091)] [2020-06-04 07:40:27,707] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.10:8091 [2020-06-04 07:40:27,721] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.10:8091 is running [2020-06-04 07:40:27,733] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.122:8091 [2020-06-04 07:40:27,745] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.122:8091 is running [2020-06-04 07:40:27,757] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.130:8091 [2020-06-04 07:40:27,769] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.130:8091 is running [2020-06-04 07:40:27,782] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.137:8091 [2020-06-04 07:40:27,807] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.137:8091 is running [2020-06-04 07:40:27,819] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.163:8091 [2020-06-04 07:40:27,831] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.163:8091 is running [2020-06-04 07:40:27,843] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.185:8091 [2020-06-04 07:40:27,854] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.185:8091 is running [2020-06-04 07:40:27,870] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.188:8091 [2020-06-04 07:40:27,884] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.188:8091 is running [2020-06-04 07:40:27,907] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.108.238:8091 [2020-06-04 07:40:27,920] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.108.238:8091 is running [2020-06-04 07:40:27,921] - [basetestcase] [139645367297856] - INFO - initializing cluster [2020-06-04 07:40:27,925] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:40:28,024] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:40:28,362] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:28,859] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:28,892] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:40:28,993] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:40:29,315] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:29,813] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:29,911] - [remote_util] [139645367297856] - INFO - 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:40:30,186] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:40:30,187] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: systemctl stop couchbase-server.service [2020-06-04 07:40:31,913] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:31,914] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:40:32,032] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:40:32,052] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:32,053] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:40:32,136] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:32,140] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:40:32,241] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:40:32,555] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:33,032] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:33,126] - [remote_util] [139645367297856] - INFO - 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:40:33,127] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:40:33,239] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:40:33,447] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:40:33,447] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: systemctl start couchbase-server.service [2020-06-04 07:40:33,484] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:33,484] - [remote_util] [139645367297856] - INFO - 172.23.105.10:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:40:38,490] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:40:38,530] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:38,533] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 22946 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:40:38,534] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:40:38,625] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:40:38,743] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:40:38,843] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:40:39,109] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:39,581] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:39,610] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:40:39,711] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:40:39,975] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:40,380] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:40,462] - [remote_util] [139645367297856] - INFO - 172.23.105.122 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:40:40,713] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:40:40,713] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: systemctl stop couchbase-server.service [2020-06-04 07:40:42,140] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:42,143] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:40:42,147] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:40:42,167] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:42,167] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:40:42,251] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:42,254] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:40:42,354] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:40:42,619] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:43,070] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:43,154] - [remote_util] [139645367297856] - INFO - 172.23.105.122 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:40:43,155] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:40:43,257] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:40:43,517] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:40:43,518] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: systemctl start couchbase-server.service [2020-06-04 07:40:43,539] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:43,539] - [remote_util] [139645367297856] - INFO - 172.23.105.122:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:40:48,546] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:40:48,573] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:48,574] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 21692 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:40:48,574] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:40:48,669] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:40:48,787] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:40:48,954] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:40:49,241] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:49,748] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:49,785] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:40:49,885] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:40:50,222] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:50,667] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:50,766] - [remote_util] [139645367297856] - INFO - 172.23.105.130 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:40:51,039] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:40:51,040] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: systemctl stop couchbase-server.service [2020-06-04 07:40:52,617] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:52,617] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:40:52,682] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:40:52,702] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:52,702] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:40:52,787] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:52,791] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:40:52,926] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:40:53,209] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:40:53,685] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:40:53,778] - [remote_util] [139645367297856] - INFO - 172.23.105.130 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:40:53,779] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:40:53,885] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:40:54,159] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:40:54,160] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: systemctl start couchbase-server.service [2020-06-04 07:40:54,202] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:54,202] - [remote_util] [139645367297856] - INFO - 172.23.105.130:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:40:59,208] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:40:59,245] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:40:59,245] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 99685 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:40:59,245] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:40:59,316] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:40:59,434] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:40:59,603] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:40:59,966] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:00,456] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:00,500] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:41:00,672] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:41:01,231] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:01,703] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:02,020] - [remote_util] [139645367297856] - INFO - 172.23.105.137 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:02,161] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:02,162] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: systemctl stop couchbase-server.service [2020-06-04 07:41:03,748] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:03,748] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:41:03,863] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:41:03,903] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:03,903] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:41:03,987] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:03,998] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:41:04,135] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:41:04,472] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:05,212] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:05,322] - [remote_util] [139645367297856] - INFO - 172.23.105.137 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:05,322] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:05,388] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:41:05,605] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:05,606] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: systemctl start couchbase-server.service [2020-06-04 07:41:05,653] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:05,654] - [remote_util] [139645367297856] - INFO - 172.23.105.137:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:41:10,659] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:41:10,700] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:10,701] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 13821 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:41:10,701] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:10,816] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:41:10,934] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:41:11,033] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:41:11,302] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:11,770] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:11,809] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:41:11,911] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:41:12,153] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:12,556] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:12,640] - [remote_util] [139645367297856] - INFO - 172.23.105.163 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:12,895] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:12,896] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: systemctl stop couchbase-server.service [2020-06-04 07:41:14,372] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:14,373] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:41:14,437] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:41:14,457] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:14,458] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:41:14,540] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:14,544] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:41:14,643] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:41:14,922] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:15,391] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:15,475] - [remote_util] [139645367297856] - INFO - 172.23.105.163 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:15,476] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:15,578] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:41:15,828] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:15,830] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: systemctl start couchbase-server.service [2020-06-04 07:41:15,852] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:15,853] - [remote_util] [139645367297856] - INFO - 172.23.105.163:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:41:20,859] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:41:20,881] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:20,881] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 11504 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:41:20,882] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:20,974] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:41:21,093] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:41:21,194] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:41:21,474] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:21,926] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:21,965] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:41:22,065] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:41:22,323] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:22,802] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:22,885] - [remote_util] [139645367297856] - INFO - 172.23.105.185 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:23,135] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:23,136] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: systemctl stop couchbase-server.service [2020-06-04 07:41:24,614] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:24,615] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:41:24,729] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:41:24,752] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:24,752] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:41:24,835] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:24,844] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:41:24,944] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:41:25,179] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:25,572] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:25,659] - [remote_util] [139645367297856] - INFO - 172.23.105.185 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:25,660] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:25,763] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:41:26,022] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:26,023] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: systemctl start couchbase-server.service [2020-06-04 07:41:26,048] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:26,049] - [remote_util] [139645367297856] - INFO - 172.23.105.185:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:41:31,050] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:41:31,072] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:31,073] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 24763 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:41:31,073] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:31,170] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:41:31,289] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:41:31,594] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:41:32,144] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:32,621] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:32,666] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:41:32,940] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:41:33,403] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:33,839] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:33,926] - [remote_util] [139645367297856] - INFO - 172.23.105.188 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:34,062] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:34,063] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: systemctl stop couchbase-server.service [2020-06-04 07:41:35,941] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:35,941] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:41:36,056] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:41:36,070] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:36,071] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:41:36,136] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:36,140] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:41:36,408] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:41:36,862] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:37,298] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:37,402] - [remote_util] [139645367297856] - INFO - 172.23.105.188 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:37,402] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:37,466] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:41:37,621] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:37,621] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: systemctl start couchbase-server.service [2020-06-04 07:41:37,659] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:37,659] - [remote_util] [139645367297856] - INFO - 172.23.105.188:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:41:42,662] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:41:42,703] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:42,703] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 11718 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:41:42,703] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:42,794] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:41:42,913] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:41:43,012] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:41:43,189] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:43,664] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:43,709] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:41:43,808] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:41:43,985] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:44,443] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:44,527] - [remote_util] [139645367297856] - INFO - 172.23.108.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:44,787] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:44,789] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: systemctl stop couchbase-server.service [2020-06-04 07:41:46,466] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:46,467] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:41:46,582] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:41:46,604] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:46,605] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:41:46,689] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:46,697] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:41:46,798] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:41:46,965] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:41:47,471] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:41:47,556] - [remote_util] [139645367297856] - INFO - 172.23.108.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:41:47,556] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:47,661] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:41:47,891] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:41:47,892] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: systemctl start couchbase-server.service [2020-06-04 07:41:47,915] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:47,916] - [remote_util] [139645367297856] - INFO - 172.23.108.238:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:41:52,921] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:41:52,959] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:41:52,959] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 2780 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:41:52,960] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:41:53,023] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:41:53,137] - [basetestcase] [139645367297856] - INFO - sleep for 10 secs. ... [2020-06-04 07:42:03,454] - [task] [139642271692544] - INFO - server: ip:172.23.105.10 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:03,467] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.10', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.10', 'uptime': u'88', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.105.10:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3629174784, 'memoryTotal': 4219441152, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:03,478] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.10:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:03,479] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2156 [2020-06-04 07:42:03,492] - [rest_client] [139642271692544] - INFO - /node/controller/setupServices params on 172.23.105.10:8091:services=kv&password=password&user=Administrator&hostname=172.23.105.10 [2020-06-04 07:42:03,506] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.10:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:03,558] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:42:03,657] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.10 [2020-06-04 07:42:03,974] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:04,470] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:04,497] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:04,548] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:04,560] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.10:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:04,571] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.10:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:04,571] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:04,631] - [task] [139642271692544] - INFO - server: ip:172.23.105.122 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:04,651] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.122', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.122', 'uptime': u'77', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.105.122:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3821944832, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:04,662] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.122:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:04,664] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2217 [2020-06-04 07:42:04,678] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.122:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:04,736] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:42:04,837] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.122 [2020-06-04 07:42:05,104] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:05,573] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:05,599] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.122: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:05,642] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:05,657] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.122:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:05,668] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.122:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:05,668] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:05,723] - [task] [139642271692544] - INFO - server: ip:172.23.105.130 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:05,740] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.130', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.130', 'uptime': u'68', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.130:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3681243136, 'memoryTotal': 4201844736, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:05,757] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.130:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:05,758] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2147 [2020-06-04 07:42:05,773] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.130:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:05,834] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:42:05,971] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.130 [2020-06-04 07:42:06,272] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:06,709] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:06,744] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.130: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:06,788] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:06,799] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.130:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:06,810] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.130:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:06,810] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:06,886] - [task] [139642271692544] - INFO - server: ip:172.23.105.137 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:06,904] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.137', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.137', 'uptime': u'58', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.105.137:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3715923968, 'memoryTotal': 4219441152, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:06,930] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.137:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:06,931] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2156 [2020-06-04 07:42:06,957] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.137:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:07,017] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:42:07,153] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.137 [2020-06-04 07:42:07,506] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:07,971] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:08,006] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.137: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:08,243] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:08,261] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.137:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:08,275] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.137:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:08,275] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:08,344] - [task] [139642271692544] - INFO - server: ip:172.23.105.163 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:08,362] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.163', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.163', 'uptime': u'47', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.105.163:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3842228224, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:08,373] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.163:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:08,375] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2217 [2020-06-04 07:42:08,388] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.163:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:08,448] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:42:08,549] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.163 [2020-06-04 07:42:08,849] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:09,318] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:09,354] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.163: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:09,395] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:09,405] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.163:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:09,416] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.163:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:09,416] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:09,481] - [task] [139642271692544] - INFO - server: ip:172.23.105.185 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:09,494] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.185', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.185', 'uptime': u'37', 'mcdMemoryReserved': 3310, 'storageTotalRam': 4138, 'hostname': u'172.23.105.185:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3801735168, 'memoryTotal': 4339109888, 'memoryQuota': 1817, 'mcdMemoryAllocated': 3310, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:09,505] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.185:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:09,506] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2217 [2020-06-04 07:42:09,520] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.185:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:09,572] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:42:09,670] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.185 [2020-06-04 07:42:09,944] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:10,364] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:10,392] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.185: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:10,427] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:10,445] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.185:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:10,463] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.185:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:10,464] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:10,520] - [task] [139642271692544] - INFO - server: ip:172.23.105.188 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:10,534] - [task] [139642271692544] - INFO - {'ip': u'172.23.105.188', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.188', 'uptime': u'26', 'mcdMemoryReserved': 3245, 'storageTotalRam': 4056, 'hostname': u'172.23.105.188:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3744636928, 'memoryTotal': 4254031872, 'memoryQuota': 1768, 'mcdMemoryAllocated': 3245, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:10,553] - [rest_client] [139642271692544] - ERROR - GET http://172.23.105.188:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:10,554] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2174 [2020-06-04 07:42:10,573] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.105.188:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:10,669] - [remote_util] [139642271692544] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:42:10,902] - [remote_util] [139642271692544] - INFO - Connected to 172.23.105.188 [2020-06-04 07:42:11,396] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:11,843] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:11,882] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.105.188: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:11,920] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:11,933] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.188:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:11,951] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.105.188:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:11,952] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:12,022] - [task] [139642271692544] - INFO - server: ip:172.23.108.238 port:8091 ssh_username:root, nodes/self [2020-06-04 07:42:12,036] - [task] [139642271692544] - INFO - {'ip': u'172.23.108.238', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'18', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3778, 'hostname': u'172.23.108.238:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.0-7737-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3397058560, 'memoryTotal': 3961597952, 'memoryQuota': 1601, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-06-04 07:42:12,053] - [rest_client] [139642271692544] - ERROR - GET http://172.23.108.238:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-06-04 07:42:12,055] - [rest_client] [139642271692544] - INFO - pools/default params : memoryQuota=2024 [2020-06-04 07:42:12,073] - [rest_client] [139642271692544] - INFO - settings/web params on 172.23.108.238:8091:username=Administrator&password=password&port=8091 [2020-06-04 07:42:12,132] - [remote_util] [139642271692544] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:42:12,265] - [remote_util] [139642271692544] - INFO - Connected to 172.23.108.238 [2020-06-04 07:42:12,489] - [remote_util] [139642271692544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:12,988] - [remote_util] [139642271692544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:13,016] - [remote_util] [139642271692544] - INFO - running command.raw on 172.23.108.238: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:13,058] - [remote_util] [139642271692544] - INFO - command executed successfully [2020-06-04 07:42:13,074] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.108.238:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:13,085] - [rest_client] [139642271692544] - INFO - /diag/eval status on 172.23.108.238:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2020-06-04 07:42:13,086] - [rest_client] [139642271692544] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:13,157] - [basetestcase] [139645367297856] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.10 **** [2020-06-04 07:42:13,210] - [rest_client] [139645367297856] - ERROR - DELETE http://172.23.105.10:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2020-06-04 07:42:13,211] - [internal_user] [139645367297856] - INFO - Exception while deleting user. Exception is -"User was not found." [2020-06-04 07:42:13,300] - [basetestcase] [139645367297856] - INFO - sleep for 5 secs. ... [2020-06-04 07:42:18,306] - [basetestcase] [139645367297856] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2020-06-04 07:42:18,344] - [basetestcase] [139645367297856] - INFO - done initializing cluster [2020-06-04 07:42:19,132] - [task] [139642271692544] - INFO - adding node 172.23.105.122:8091 to cluster [2020-06-04 07:42:19,133] - [rest_client] [139642271692544] - INFO - adding remote node @172.23.105.122:8091 to this cluster @172.23.105.10:8091 [2020-06-04 07:42:20,602] - [task] [139642271692544] - INFO - adding node 172.23.105.130:8091 to cluster [2020-06-04 07:42:20,603] - [rest_client] [139642271692544] - INFO - adding remote node @172.23.105.130:8091 to this cluster @172.23.105.10:8091 [2020-06-04 07:42:22,306] - [task] [139642271692544] - INFO - adding node 172.23.105.137:8091 to cluster [2020-06-04 07:42:22,306] - [rest_client] [139642271692544] - INFO - adding remote node @172.23.105.137:8091 to this cluster @172.23.105.10:8091 [2020-06-04 07:42:24,165] - [task] [139642271692544] - INFO - adding node 172.23.105.163:8091 to cluster [2020-06-04 07:42:24,167] - [rest_client] [139642271692544] - INFO - adding remote node @172.23.105.163:8091 to this cluster @172.23.105.10:8091 [2020-06-04 07:42:25,785] - [rest_client] [139642271692544] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.163,ns_1@172.23.105.10,ns_1@172.23.105.137,ns_1@172.23.105.122'} [2020-06-04 07:42:25,803] - [rest_client] [139642271692544] - INFO - rebalance operation started [2020-06-04 07:42:25,816] - [rest_client] [139642271692544] - INFO - rebalance percentage : 0.00 % [2020-06-04 07:42:25,817] - [task] [139642271692544] - INFO - Rebalance - status: running, progress: 0.00% [2020-06-04 07:42:35,855] - [task] [139642271692544] - INFO - Rebalance - status: none, progress: 100.00% [2020-06-04 07:42:35,872] - [task] [139642271692544] - INFO - rebalancing was completed with progress: 100% in 10.0678489208 sec [2020-06-04 07:42:35,876] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:42:35,978] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:42:36,336] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:36,834] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:36,861] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-06-04 07:42:36,912] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:42:36,913] - [basetestcase] [139645367297856] - INFO - Enabled diag/eval for non-local hosts from 172.23.105.10 [2020-06-04 07:42:36,914] - [basetestcase] [139645367297856] - INFO - ============== basetestcase setup was finished for test #8 suite_tearDown ============== [2020-06-04 07:42:36,971] - [basetestcase] [139645367297856] - INFO - ------- Cluster statistics ------- [2020-06-04 07:42:36,971] - [basetestcase] [139645367297856] - INFO - 172.23.105.130:8091 => {'swap_mem_used': 44040192, 'cpu_utilization': 0.5012531328320802, 'mem_free': 3636834304, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} [2020-06-04 07:42:36,972] - [basetestcase] [139645367297856] - INFO - 172.23.105.163:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3760173056, 'swap_mem_total': 3758092288, 'mem_total': 4339109888, 'services': [u'index', u'n1ql']} [2020-06-04 07:42:36,972] - [basetestcase] [139645367297856] - INFO - 172.23.105.10:8091 => {'swap_mem_used': 38535168, 'cpu_utilization': 0.75, 'mem_free': 3579789312, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'kv']} [2020-06-04 07:42:36,973] - [basetestcase] [139645367297856] - INFO - 172.23.105.137:8091 => {'swap_mem_used': 786432, 'cpu_utilization': 0.9950248756218906, 'mem_free': 3664150528, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'eventing']} [2020-06-04 07:42:36,973] - [basetestcase] [139645367297856] - INFO - 172.23.105.122:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0, 'mem_free': 3779125248, 'swap_mem_total': 3758092288, 'mem_total': 4339109888, 'services': [u'kv']} [2020-06-04 07:42:36,974] - [basetestcase] [139645367297856] - INFO - --- End of cluster statistics --- [2020-06-04 07:42:36,977] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:42:37,078] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:42:37,379] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:42:37,815] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:42:42,509] - [basetestcase] [139645367297856] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.163 port:8091 ssh_username:root] [2020-06-04 07:42:42,538] - [basetestcase] [139645367297856] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:42:42,552] - [rest_client] [139645367297856] - INFO - settings/indexes params : storageMode=plasma [2020-06-04 07:42:42,569] - [eventing_base] [139645367297856] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2020-06-04 07:42:42,569] - [rest_client] [139645367297856] - INFO - pools/default params : memoryQuota=330 [2020-06-04 07:42:42,592] - [rest_client] [139645367297856] - INFO - pools/default params : indexMemoryQuota=256 [2020-06-04 07:42:42,610] - [rest_client] [139645367297856] - INFO - pools/default params : eventingMemoryQuota=512 [2020-06-04 07:42:42,661] - [basetestcase] [139645367297856] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.163 port:8091 ssh_username:root] [2020-06-04 07:42:42,662] - [rest_client] [139645367297856] - INFO - pools/default params : memoryQuota=700 [2020-06-04 07:42:42,677] - [eventing_rebalance] [139645367297856] - INFO - 100 [2020-06-04 07:42:42,968] - [rest_client] [139642271692544] - INFO - http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=100 [2020-06-04 07:42:42,983] - [rest_client] [139642271692544] - INFO - 0.01 seconds to create bucket src_bucket [2020-06-04 07:42:42,983] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:43,313] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:43,485] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:43,667] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:44,161] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:44,325] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:44,505] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:44,636] - [task] [139642271692544] - WARNING - vbucket map not ready after try 0 [2020-06-04 07:42:44,638] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:45,113] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:45,288] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:45,455] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:45,695] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:46,115] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:46,294] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:46,423] - [task] [139642271692544] - WARNING - vbucket map not ready after try 1 [2020-06-04 07:42:46,424] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:46,656] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:47,081] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:47,244] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:47,494] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:47,653] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:48,075] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:48,218] - [task] [139642271692544] - WARNING - vbucket map not ready after try 2 [2020-06-04 07:42:48,219] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:48,445] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:48,625] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:48,782] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:49,261] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:49,436] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:49,605] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:49,744] - [task] [139642271692544] - WARNING - vbucket map not ready after try 3 [2020-06-04 07:42:49,745] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:50,214] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:50,391] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:50,562] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:50,795] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:51,219] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:51,379] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:51,502] - [task] [139642271692544] - WARNING - vbucket map not ready after try 4 [2020-06-04 07:42:51,503] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:51,745] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:51,899] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:52,326] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:52,556] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 src_bucket [2020-06-04 07:42:52,736] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 src_bucket [2020-06-04 07:42:52,902] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 src_bucket [2020-06-04 07:42:53,290] - [task] [139642271692544] - WARNING - vbucket map not ready after try 5 [2020-06-04 07:42:54,377] - [rest_client] [139642271692544] - INFO - http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=100 [2020-06-04 07:42:54,395] - [rest_client] [139642271692544] - INFO - 0.02 seconds to create bucket dst_bucket [2020-06-04 07:42:54,395] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:54,687] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:42:54,843] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:42:55,030] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:42:55,522] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:42:55,705] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:42:55,860] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:42:56,017] - [task] [139642271692544] - WARNING - vbucket map not ready after try 0 [2020-06-04 07:42:56,018] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:56,274] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:42:56,695] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:42:56,859] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:42:57,098] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:42:57,268] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:42:57,437] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:42:57,819] - [task] [139642271692544] - WARNING - vbucket map not ready after try 1 [2020-06-04 07:42:57,820] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:58,055] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:42:58,239] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:42:58,667] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:42:58,922] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:42:59,114] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:42:59,285] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:42:59,433] - [task] [139642271692544] - WARNING - vbucket map not ready after try 2 [2020-06-04 07:42:59,433] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:42:59,923] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:43:00,083] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:43:00,280] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:43:00,546] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:43:00,967] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:43:01,135] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:43:01,260] - [task] [139642271692544] - WARNING - vbucket map not ready after try 3 [2020-06-04 07:43:01,261] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:43:01,509] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:43:01,944] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:43:02,122] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:43:02,375] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:43:02,539] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:43:02,710] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:43:02,832] - [task] [139642271692544] - WARNING - vbucket map not ready after try 4 [2020-06-04 07:43:02,833] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops [2020-06-04 07:43:03,344] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:43:03,543] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:43:03,723] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:43:04,226] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 dst_bucket [2020-06-04 07:43:04,415] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 dst_bucket [2020-06-04 07:43:04,592] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 dst_bucket [2020-06-04 07:43:04,715] - [task] [139642271692544] - WARNING - vbucket map not ready after try 5 [2020-06-04 07:43:05,800] - [rest_client] [139642271692544] - INFO - http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=400 [2020-06-04 07:43:05,816] - [rest_client] [139642271692544] - INFO - 0.02 seconds to create bucket metadata [2020-06-04 07:43:05,817] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 07:43:06,371] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:06,558] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:06,737] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:06,976] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:07,151] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:07,582] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:07,723] - [task] [139642271692544] - WARNING - vbucket map not ready after try 0 [2020-06-04 07:43:07,724] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 07:43:07,962] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:08,141] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:08,326] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:08,810] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:09,011] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:09,193] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:09,319] - [task] [139642271692544] - WARNING - vbucket map not ready after try 1 [2020-06-04 07:43:09,319] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 07:43:09,834] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:10,000] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:10,183] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:10,439] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:10,872] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:11,047] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:11,173] - [task] [139642271692544] - WARNING - vbucket map not ready after try 2 [2020-06-04 07:43:11,174] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 07:43:11,424] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:11,601] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:12,035] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:12,302] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:12,478] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:12,653] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:13,030] - [task] [139642271692544] - WARNING - vbucket map not ready after try 3 [2020-06-04 07:43:13,031] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 07:43:13,265] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:13,445] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:13,634] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:14,149] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:14,326] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:14,506] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:14,630] - [task] [139642271692544] - WARNING - vbucket map not ready after try 4 [2020-06-04 07:43:14,630] - [bucket_helper] [139642271692544] - INFO - waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops [2020-06-04 07:43:14,870] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:15,318] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:15,497] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:15,745] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.10:11210 metadata [2020-06-04 07:43:15,934] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.122:11210 metadata [2020-06-04 07:43:16,362] - [data_helper] [139642271692544] - INFO - creating direct client 172.23.105.130:11210 metadata [2020-06-04 07:43:16,514] - [task] [139642271692544] - WARNING - vbucket map not ready after try 5 [2020-06-04 07:43:16,828] - [basetestcase] [139645367297856] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:43:16,873] - [eventing_base] [139645367297856] - INFO - Go routine dumps for Node 172.23.105.137 is goroutine profile: total 76 17 @ 0x913f1a 0x923c30 0xb86b3b 0x9417b1 # 0xb86b3a net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 11 @ 0x913f1a 0x923c30 0xb85b33 0x9417b1 # 0xb85b32 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 9 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2cb9c 0xb2ed24 0xb2e802 0xb2eae3 0xb7c0d1 0xb7c01d 0xb8827c 0x9e5d52 0x9e5a5d 0x9e54a8 0xc4b91e 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb2ed23 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb2e801 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb2eae2 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb7c0d0 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb7c01c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb8827b net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e5d51 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e5a5c encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e54a7 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc4b91d github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:288 6 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2c00a 0xb85575 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2c009 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb85574 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 3 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb6d188 0xa2bebe 0xa2c00a 0xb72413 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb6d187 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2c009 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb72412 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 2 @ 0x913f1a 0x923c30 0x1126a1c 0x9417b1 # 0x1126a1b github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x8f6ab2 0x9289c6 0xd97232 0x9417b1 # 0x9289c5 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd97231 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c548 0xc005dd 0x9e5d52 0x9e5a5d 0x9e54a8 0xbff15f 0xbfce27 0xbfc71a 0xbfbd87 0xc00b49 0xc013ac 0x11268ad 0x115b93e 0x1179290 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c547 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xc005dc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e5d51 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e5a5c encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e54a7 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbff15e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbfce26 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbfc719 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbfbd86 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xc00b48 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xc013ab github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x11268ac github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x115b93d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:444 # 0x117928f github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xa2c548 0xc005dd 0x9e5d52 0x9e5a5d 0x9e54a8 0xbff15f 0xbfce27 0xbfc71a 0xbfbd87 0xc00b49 0xc013ac 0xc44880 0xc45f35 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa2c547 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xc005dc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e5d51 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e5a5c encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e54a7 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbff15e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbfce26 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbfc719 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbfbd86 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xc00b48 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xc013ab github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc4487f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc45f34 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb6ccda 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb6ccd9 net/http.(*connReader).backgroundRead+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:668 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2cb9c 0xb2ed24 0xb2e802 0xb2eae3 0xb7c0d1 0xb7c01d 0xb8827c 0xa2bebe 0xa2cb9c 0xa2d05b 0xc8a507 0xc89e4e 0xd53783 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb2ed23 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb2e801 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb2eae2 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb7c0d0 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb7c01c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb8827b net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa2d05a bufio.(*Reader).ReadBytes+0x6a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:419 # 0xc8a506 github.com/couchbase/eventing/dcp.(*Client).runObserveStreamingEndpoint+0x526 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:409 # 0xc89e4d github.com/couchbase/eventing/dcp.(*Client).RunObservePool+0xdd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:345 # 0xd53782 github.com/couchbase/eventing/util.(*serviceNotifierInstance).RunPoolObserver+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/services_notifier.go:79 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9aa7ed 0xa73cef 0xa84fca 0xb84ab6 0xa2bebe 0xa2cb9c 0xb2ed24 0xb2e802 0xb2eae3 0xb7c0d1 0xb7c01d 0xb8827c 0xa2bebe 0xa2cb9c 0xa2d05b 0xc8a507 0xc89f96 0xd53953 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9aa7ec internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa73cee net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa84fc9 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb84ab5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb2ed23 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb2e801 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb2eae2 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb7c0d0 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb7c01c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb8827b net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0xa2bebd bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2cb9b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa2d05a bufio.(*Reader).ReadBytes+0x6a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:419 # 0xc8a506 github.com/couchbase/eventing/dcp.(*Client).runObserveStreamingEndpoint+0x526 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:409 # 0xc89f95 github.com/couchbase/eventing/dcp.(*Client).RunObserveNodeServices+0xf5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:361 # 0xd53952 github.com/couchbase/eventing/util.(*serviceNotifierInstance).RunServicesObserver+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/services_notifier.go:88 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9abd88 0xa74602 0xa8e94e 0xa8cf29 0xb775cf 0xaf2407 0xb76365 0xb7697b 0xb7726e 0x11799f6 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9abd87 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa74601 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa8e94d net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa8cf28 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb775ce net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaf2406 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb76364 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb7697a net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb7726d net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x11799f5 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:227 1 @ 0x913f1a 0x90f23a 0x90e8b7 0x9a990b 0x9a998d 0x9abd88 0xa74602 0xa8e94e 0xa8cf29 0xb775cf 0xb76365 0xb760b9 0x1179477 0x9417b1 # 0x90e8b6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a990a internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a998c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9abd87 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa74601 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa8e94d net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa8cf28 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb775ce net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb76364 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb760b8 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x1179476 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:185 1 @ 0x913f1a 0x913fce 0x8eb992 0x8eb64b 0x118ea55 0x119a14c 0x913ac2 0x9417b1 # 0x118ea54 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x119a14b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x913ac1 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x913f1a 0x913fce 0x8eb992 0x8eb64b 0x1194f53 0x9417b1 # 0x1194f52 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:84 1 @ 0x913f1a 0x913fce 0x930476 0xd52123 0x9417b1 # 0x930475 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xd52122 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x913f1a 0x923c30 0x115c537 0x11612c2 0x1126ce4 0x112610b 0x1126503 0x11276a2 0x93f06b 0x9911d9 0x990754 0xbfb4ae 0x9417b1 # 0x115c536 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:541 # 0x11612c1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1126ce3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x112610a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1126502 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93f06a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x9911d8 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x990753 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbfb4ad net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x913f1a 0x923c30 0x115c537 0x1161a52 0x1126b34 0x112610b 0x11263a3 0x11275d2 0x93f06b 0x9911d9 0x990754 0xbfb4ae 0x9417b1 # 0x115c536 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:541 # 0x1161a51 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1126b33 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x112610a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11263a2 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93f06a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x9911d8 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x990753 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbfb4ad net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x913f1a 0x923c30 0x1192f5a 0x9417b1 # 0x1192f59 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).watchBucketChanges+0x409 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/util.go:244 1 @ 0x913f1a 0x923c30 0xb9e321 0x9417b1 # 0xb9e320 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x913f1a 0x923c30 0xb9e78f 0x9417b1 # 0xb9e78e github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x1179d2c 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x1179d2b github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0xab /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:241 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x1179ecc 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x1179ecb github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0xab /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:252 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x117a06c 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x117a06b github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0xab /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:263 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119a896 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119a895 main.main.func1+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119a9d6 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119a9d5 main.main.func2+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119ab16 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119ab15 main.main.func3+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119ac56 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119ac55 main.main.func4+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119ad96 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119ad95 main.main.func5+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x913f1a 0x923c30 0xc48d99 0xc48a68 0xc4953a 0x119aed6 0x9417b1 # 0xc48d98 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2e8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:305 # 0xc48a67 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:263 # 0xc49539 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x89 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:392 # 0x119aed5 main.main.func6+0xa5 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0x913f1a 0x923c30 0xd50039 0x9417b1 # 0xd50038 github.com/couchbase/eventing/util.(*ClusterInfoClient).watchClusterChanges+0x328 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:729 1 @ 0x913f1a 0x923c30 0xd72fe3 0x9417b1 # 0xd72fe2 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 1 @ 0xd8e858 0xd8e660 0xd8b1a4 0xd9696d 0xd96cf1 0x11559ed 0x117ad88 0xb73284 0xb74ef0 0xb75f9c 0xb722a1 0x9417b1 # 0xd8e857 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd8e65f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd8b1a3 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd9696c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd96cf0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11559ec github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3571 # 0x117ad87 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb73283 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb74eef net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb75f9b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb722a0 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2020-06-04 07:43:16,912] - [basetestcase] [139645367297856] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:43:16,946] - [eventing_base] [139645367297856] - INFO - Stats for Node 172.23.105.137 is [] [2020-06-04 07:43:17,011] - [basetestcase] [139645367297856] - INFO - list of eventing nodes in cluster: [ip:172.23.105.137 port:8091 ssh_username:root] [2020-06-04 07:43:17,015] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:43:17,151] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:43:17,508] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:43:17,964] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:43:18,007] - [rest_client] [139645367297856] - INFO - /diag/eval status on 172.23.105.137:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-06-04 07:43:18,008] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-06-04 07:43:18,429] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:43:18,429] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-06-04 07:43:18,468] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:43:18,997] - [eventing_base] [139645367297856] - INFO - Bucket dst_bucket DGM is 100 [2020-06-04 07:43:19,035] - [eventing_base] [139645367297856] - INFO - Bucket metadata DGM is 100 [2020-06-04 07:43:19,072] - [eventing_base] [139645367297856] - INFO - Bucket src_bucket DGM is 100 [2020-06-04 07:43:19,094] - [eventing_base] [139645367297856] - INFO - number of documents in metadata bucket 0 [2020-06-04 07:43:19,144] - [basetestcase] [139645367297856] - INFO - ------- Cluster statistics ------- [2020-06-04 07:43:19,145] - [basetestcase] [139645367297856] - INFO - 172.23.105.130:8091 => {'swap_mem_used': 44040192, 'cpu_utilization': 4.030226700251889, 'mem_free': 3525038080, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} [2020-06-04 07:43:19,145] - [basetestcase] [139645367297856] - INFO - 172.23.105.163:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3652587520, 'swap_mem_total': 3758092288, 'mem_total': 4339109888, 'services': [u'index', u'n1ql']} [2020-06-04 07:43:19,145] - [basetestcase] [139645367297856] - INFO - 172.23.105.10:8091 => {'swap_mem_used': 38535168, 'cpu_utilization': 3.5, 'mem_free': 3311140864, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'kv']} [2020-06-04 07:43:19,146] - [basetestcase] [139645367297856] - INFO - 172.23.105.137:8091 => {'swap_mem_used': 786432, 'cpu_utilization': 13.29923273657289, 'mem_free': 3625947136, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'eventing']} [2020-06-04 07:43:19,146] - [basetestcase] [139645367297856] - INFO - 172.23.105.122:8091 => {'swap_mem_used': 0, 'cpu_utilization': 3.759398496240602, 'mem_free': 3657371648, 'swap_mem_total': 3758092288, 'mem_total': 4339109888, 'services': [u'kv']} [2020-06-04 07:43:19,147] - [basetestcase] [139645367297856] - INFO - --- End of cluster statistics --- [2020-06-04 07:43:19,148] - [basetestcase] [139645367297856] - INFO - ============== basetestcase cleanup was started for test #8 suite_tearDown ============== [2020-06-04 07:43:19,290] - [bucket_helper] [139645367297856] - INFO - deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket'] on 172.23.105.10 [2020-06-04 07:43:19,290] - [bucket_helper] [139645367297856] - INFO - remove bucket dst_bucket ... [2020-06-04 07:43:19,705] - [bucket_helper] [139645367297856] - INFO - deleted bucket : dst_bucket from 172.23.105.10 [2020-06-04 07:43:19,705] - [bucket_helper] [139645367297856] - INFO - waiting for bucket deletion to complete.... [2020-06-04 07:43:19,765] - [rest_client] [139645367297856] - INFO - node 172.23.105.10 existing buckets : [u'metadata', u'src_bucket'] [2020-06-04 07:43:19,768] - [bucket_helper] [139645367297856] - INFO - remove bucket metadata ... [2020-06-04 07:43:20,181] - [bucket_helper] [139645367297856] - INFO - deleted bucket : metadata from 172.23.105.10 [2020-06-04 07:43:20,181] - [bucket_helper] [139645367297856] - INFO - waiting for bucket deletion to complete.... [2020-06-04 07:43:20,209] - [rest_client] [139645367297856] - INFO - node 172.23.105.10 existing buckets : [u'src_bucket'] [2020-06-04 07:43:20,210] - [bucket_helper] [139645367297856] - INFO - remove bucket src_bucket ... [2020-06-04 07:43:20,608] - [bucket_helper] [139645367297856] - INFO - deleted bucket : src_bucket from 172.23.105.10 [2020-06-04 07:43:20,609] - [bucket_helper] [139645367297856] - INFO - waiting for bucket deletion to complete.... [2020-06-04 07:43:20,620] - [rest_client] [139645367297856] - INFO - node 172.23.105.10 existing buckets : [] [2020-06-04 07:43:20,621] - [bucket_helper] [139645367297856] - INFO - sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket']) were deleted completely. [2020-06-04 07:43:22,896] - [cluster_helper] [139645367297856] - INFO - rebalancing all nodes in order to remove nodes [2020-06-04 07:43:22,908] - [rest_client] [139645367297856] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.163,ns_1@172.23.105.137,ns_1@172.23.105.122', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.130,ns_1@172.23.105.163,ns_1@172.23.105.10,ns_1@172.23.105.137,ns_1@172.23.105.122'} [2020-06-04 07:43:22,922] - [rest_client] [139645367297856] - INFO - rebalance operation started [2020-06-04 07:43:22,949] - [rest_client] [139645367297856] - INFO - rebalance percentage : 0.00 % [2020-06-04 07:43:32,978] - [rest_client] [139645367297856] - INFO - rebalance percentage : 70.00 % [2020-06-04 07:43:43,010] - [rest_client] [139645367297856] - INFO - rebalance percentage : 70.00 % [2020-06-04 07:44:03,050] - [rest_client] [139645367297856] - INFO - rebalance progress took 40.13 seconds [2020-06-04 07:44:03,050] - [rest_client] [139645367297856] - INFO - sleep for 10 seconds after rebalance... [2020-06-04 07:44:13,207] - [cluster_helper] [139645367297856] - INFO - removed all the nodes from cluster associated with ip:172.23.105.10 port:8091 ssh_username:root ? [(u'ns_1@172.23.105.130', 8091), (u'ns_1@172.23.105.163', 8091), (u'ns_1@172.23.105.137', 8091), (u'ns_1@172.23.105.122', 8091)] [2020-06-04 07:44:13,228] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.10:8091 [2020-06-04 07:44:13,248] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.10:8091 is running [2020-06-04 07:44:13,271] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.122:8091 [2020-06-04 07:44:13,283] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.122:8091 is running [2020-06-04 07:44:13,303] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.130:8091 [2020-06-04 07:44:13,322] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.130:8091 is running [2020-06-04 07:44:14,373] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.137:8091 [2020-06-04 07:44:14,411] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.137:8091 is running [2020-06-04 07:44:14,423] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.163:8091 [2020-06-04 07:44:14,436] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.163:8091 is running [2020-06-04 07:44:14,448] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.185:8091 [2020-06-04 07:44:14,474] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.185:8091 is running [2020-06-04 07:44:14,489] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.105.188:8091 [2020-06-04 07:44:14,505] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.105.188:8091 is running [2020-06-04 07:44:14,527] - [cluster_helper] [139645367297856] - INFO - waiting for ns_server @ 172.23.108.238:8091 [2020-06-04 07:44:14,549] - [cluster_helper] [139645367297856] - INFO - ns_server @ 172.23.108.238:8091 is running [2020-06-04 07:44:14,549] - [basetestcase] [139645367297856] - INFO - ============== basetestcase cleanup was finished for test #8 suite_tearDown ============== [2020-06-04 07:44:14,567] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:44:14,740] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:44:15,139] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:15,676] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:15,720] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:44:15,892] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:44:16,253] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:16,768] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:16,839] - [remote_util] [139645367297856] - INFO - 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:16,978] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:16,978] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: systemctl stop couchbase-server.service [2020-06-04 07:44:18,662] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:18,664] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:44:18,779] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:44:18,801] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:18,801] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:44:18,887] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:18,893] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.10 with username:root [2020-06-04 07:44:18,992] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.10 [2020-06-04 07:44:19,313] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:19,810] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:19,901] - [remote_util] [139645367297856] - INFO - 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:19,901] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:20,010] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:44:20,211] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:20,212] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: systemctl start couchbase-server.service [2020-06-04 07:44:20,248] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:20,248] - [remote_util] [139645367297856] - INFO - 172.23.105.10:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:44:25,254] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.10: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:44:25,293] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:25,294] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 24074 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:44:25,296] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:25,371] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:44:25,498] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:44:25,597] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:44:25,879] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:26,360] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:26,403] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:44:26,502] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:44:26,740] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:27,210] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:27,303] - [remote_util] [139645367297856] - INFO - 172.23.105.122 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:27,559] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:27,560] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: systemctl stop couchbase-server.service [2020-06-04 07:44:29,033] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:29,034] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:44:29,098] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:44:29,114] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:29,114] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:44:29,190] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:29,207] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.122 with username:root [2020-06-04 07:44:29,307] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.122 [2020-06-04 07:44:29,546] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:30,010] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:30,096] - [remote_util] [139645367297856] - INFO - 172.23.105.122 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:30,098] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:30,207] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:44:30,418] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:30,419] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: systemctl start couchbase-server.service [2020-06-04 07:44:30,442] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:30,442] - [remote_util] [139645367297856] - INFO - 172.23.105.122:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:44:35,447] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.122: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:44:35,486] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:35,487] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 22721 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:44:35,487] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:35,551] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:44:35,669] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:44:35,808] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:44:36,092] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:36,564] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:36,593] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:44:36,728] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:44:37,049] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:37,547] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:37,636] - [remote_util] [139645367297856] - INFO - 172.23.105.130 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:37,909] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:37,910] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: systemctl stop couchbase-server.service [2020-06-04 07:44:39,486] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:39,487] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:44:39,602] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:44:39,622] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:39,624] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:44:39,709] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:39,721] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.130 with username:root [2020-06-04 07:44:39,823] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.130 [2020-06-04 07:44:40,160] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:40,655] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:40,743] - [remote_util] [139645367297856] - INFO - 172.23.105.130 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:40,743] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:40,852] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:44:41,128] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:41,130] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: systemctl start couchbase-server.service [2020-06-04 07:44:41,172] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:41,173] - [remote_util] [139645367297856] - INFO - 172.23.105.130:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:44:46,176] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:44:46,215] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:46,215] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 100945 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:44:46,215] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:46,285] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:44:46,403] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:44:46,535] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:44:46,908] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:47,433] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:47,483] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:44:47,655] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:44:48,024] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:48,548] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:48,877] - [remote_util] [139645367297856] - INFO - 172.23.105.137 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:49,038] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:49,039] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: systemctl stop couchbase-server.service [2020-06-04 07:44:50,627] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:50,627] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:44:50,692] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:44:50,936] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:50,937] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:44:51,025] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:51,029] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.137 with username:root [2020-06-04 07:44:51,239] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.137 [2020-06-04 07:44:51,560] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:52,050] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:52,166] - [remote_util] [139645367297856] - INFO - 172.23.105.137 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:44:52,166] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:52,229] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:44:52,616] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:44:52,616] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: systemctl start couchbase-server.service [2020-06-04 07:44:52,657] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:52,657] - [remote_util] [139645367297856] - INFO - 172.23.105.137:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:44:57,662] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.137: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:44:57,714] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:44:57,714] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 15134 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:44:57,714] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:44:58,034] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:44:58,152] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:44:58,254] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:44:58,539] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:59,002] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:59,032] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:44:59,131] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:44:59,415] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:44:59,823] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:44:59,904] - [remote_util] [139645367297856] - INFO - 172.23.105.163 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:00,157] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:00,157] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: systemctl stop couchbase-server.service [2020-06-04 07:45:01,637] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:01,637] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:45:01,702] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:45:01,721] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:01,722] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:45:01,796] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:01,803] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.163 with username:root [2020-06-04 07:45:01,902] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.163 [2020-06-04 07:45:02,173] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:02,647] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:02,730] - [remote_util] [139645367297856] - INFO - 172.23.105.163 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:02,730] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:02,833] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:45:03,090] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:03,091] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: systemctl start couchbase-server.service [2020-06-04 07:45:03,113] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:03,113] - [remote_util] [139645367297856] - INFO - 172.23.105.163:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:45:08,118] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.163: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:45:08,156] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:08,156] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 12586 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:45:08,157] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:08,220] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:45:08,339] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:45:08,439] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:45:08,724] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:09,190] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:09,230] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:45:09,329] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:45:09,585] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:10,065] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:10,148] - [remote_util] [139645367297856] - INFO - 172.23.105.185 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:10,399] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:10,401] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: systemctl stop couchbase-server.service [2020-06-04 07:45:11,878] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:11,879] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:45:11,993] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:45:12,019] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:12,019] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:45:12,103] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:12,115] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.185 with username:root [2020-06-04 07:45:12,214] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.185 [2020-06-04 07:45:12,469] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:12,950] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:13,007] - [remote_util] [139645367297856] - INFO - 172.23.105.185 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:13,008] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:13,115] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:45:13,364] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:13,365] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: systemctl start couchbase-server.service [2020-06-04 07:45:13,385] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:13,387] - [remote_util] [139645367297856] - INFO - 172.23.105.185:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:45:18,393] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.185: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:45:18,429] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:18,430] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 25515 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:45:18,430] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:18,491] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:45:18,613] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:45:18,847] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:45:19,307] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:19,735] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:19,772] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:45:20,005] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:45:20,450] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:20,863] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:20,959] - [remote_util] [139645367297856] - INFO - 172.23.105.188 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:21,105] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:21,106] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: systemctl stop couchbase-server.service [2020-06-04 07:45:22,983] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:22,984] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:45:23,098] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:45:23,111] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:23,111] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:45:23,195] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:23,199] - [remote_util] [139645367297856] - INFO - connecting to 172.23.105.188 with username:root [2020-06-04 07:45:23,431] - [remote_util] [139645367297856] - INFO - Connected to 172.23.105.188 [2020-06-04 07:45:23,889] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:24,374] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:24,463] - [remote_util] [139645367297856] - INFO - 172.23.105.188 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:24,464] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:24,598] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:45:24,711] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:24,712] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: systemctl start couchbase-server.service [2020-06-04 07:45:24,750] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:24,751] - [remote_util] [139645367297856] - INFO - 172.23.105.188:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:45:29,757] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.105.188: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:45:29,799] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:29,799] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 12491 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:45:29,799] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:29,880] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:45:29,999] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:45:30,099] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:45:30,312] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:30,827] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:30,870] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:45:30,969] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:45:31,179] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:31,676] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:31,729] - [remote_util] [139645367297856] - INFO - 172.23.108.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:31,976] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:31,976] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: systemctl stop couchbase-server.service [2020-06-04 07:45:33,600] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:33,601] - [basetestcase] [139645367297856] - INFO - Couchbase stopped [2020-06-04 07:45:33,715] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-06-04 07:45:33,735] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:33,736] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-06-04 07:45:33,819] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:33,823] - [remote_util] [139645367297856] - INFO - connecting to 172.23.108.238 with username:root [2020-06-04 07:45:33,923] - [remote_util] [139645367297856] - INFO - Connected to 172.23.108.238 [2020-06-04 07:45:34,104] - [remote_util] [139645367297856] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-06-04 07:45:34,605] - [remote_util] [139645367297856] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2020-06-04 07:45:34,689] - [remote_util] [139645367297856] - INFO - 172.23.108.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-06-04 07:45:34,690] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:34,787] - [remote_util] [139645367297856] - INFO - Starting couchbase server [2020-06-04 07:45:35,051] - [remote_util] [139645367297856] - INFO - Running systemd command on this server [2020-06-04 07:45:35,052] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: systemctl start couchbase-server.service [2020-06-04 07:45:35,091] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:35,092] - [remote_util] [139645367297856] - INFO - 172.23.108.238:sleep for 5 secs. waiting for couchbase server to come up ... [2020-06-04 07:45:40,097] - [remote_util] [139645367297856] - INFO - running command.raw on 172.23.108.238: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2020-06-04 07:45:40,134] - [remote_util] [139645367297856] - INFO - command executed successfully [2020-06-04 07:45:40,134] - [remote_util] [139645367297856] - INFO - Couchbase server status: [' Process: 3585 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2020-06-04 07:45:40,135] - [remote_util] [139645367297856] - INFO - Checking for process beam.smp on linux [2020-06-04 07:45:40,198] - [basetestcase] [139645367297856] - INFO - Couchbase started [2020-06-04 07:45:40,313] - [basetestcase] [139645367297856] - INFO - sleep for 10 secs. ...