./testrunner -i /tmp/testexec.13301.ini -p get-cbcollect-info=True,GROUP=bucket_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_rebalance_in_with_different_topologies,doc-per-day=20,dataset=default,nodes_init=4,services_init=kv-kv-eventing-index:n1ql,groups=simple,services_in=eventing,reset_services=True,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers Test Input params: {'doc-per-day': '20', 'logs_folder': '/data/workspace/centos-p0-eventing-vset00-00-topology_bucket_op_with_timers/logs/testrunner-19-Dec-05_21-18-52/test_1', 'GROUP': 'bucket_op_with_timers', 'services_in': 'eventing', 'case_number': 1, 'handler_code': 'bucket_op_with_timers', 'reset_services': 'True', 'dataset': 'default', 'cluster_name': 'testexec.13301', 'num_nodes': 8, 'services_init': 'kv-kv-eventing-index:n1ql', 'ini': '/tmp/testexec.13301.ini', 'groups': 'simple', 'conf_file': 'eventing/eventing_topology.conf', 'get-cbcollect-info': 'True', 'spec': 'eventing_topology', 'nodes_init': '4'} Run before suite setup for eventing.eventing_rebalance.EventingRebalance.test_rebalance_in_with_different_topologies 2019-12-05 21:18:52 | INFO | MainProcess | test_thread | [eventing_base.setUp] Starting Test: test_rebalance_in_with_different_topologies 2019-12-05 21:18:52 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:18:52 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:18:53 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 6.5.0-4926-enterprise 2019-12-05 21:18:53 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'6.5.0-4926-enterprise'] 2019-12-05 21:18:53 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_rebalance_in_with_different_topologies============== 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.10:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.10:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.102:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.102:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.105:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.105:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.107:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.107:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.88:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.88:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.89:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.89:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.14:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.14:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.15:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.15:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.10:8091 => {'swap_mem_used': 118001664, 'cpu_utilization': 0.5, 'mem_free': 3688632320, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'index', u'kv', u'n1ql']} 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_rebalance_in_with_different_topologies ============== 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.10:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.10:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.102:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.102:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.105:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.105:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.107.107:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.107.107:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.88:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.88:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.97.89:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.97.89:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.14:8091 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.14:8091 is running 2019-12-05 21:18:54 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.98.15:8091 2019-12-05 21:18:55 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.98.15:8091 is running 2019-12-05 21:18:55 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_rebalance_in_with_different_topologies ============== 2019-12-05 21:18:55 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:18:55 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:18:55 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:18:56 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:18:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:18:57 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:18:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: systemctl stop couchbase-server.service 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:18:58 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:18:59 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:18:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:18:59 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:18:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: systemctl start couchbase-server.service 2019-12-05 21:19:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:00 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:00 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:19:00 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:19:01 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:19:01 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:19:02 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.102 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:02 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: systemctl stop couchbase-server.service 2019-12-05 21:19:03 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:03 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:04 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:19:04 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.102 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: systemctl start couchbase-server.service 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:19:05 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:19:06 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:19:06 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:19:07 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.105 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:07 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: systemctl stop couchbase-server.service 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:19:09 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.105 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: systemctl start couchbase-server.service 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:19:10 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:19:11 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:19:11 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:19:12 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.107 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:12 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: systemctl stop couchbase-server.service 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:19:15 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:19:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.107 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:15 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: systemctl start couchbase-server.service 2019-12-05 21:19:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:16 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:16 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:19:16 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:19:17 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:19:17 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:19:18 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.88 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:18 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: systemctl stop couchbase-server.service 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:19 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:19:20 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:19:20 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.88 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:21 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: systemctl start couchbase-server.service 2019-12-05 21:19:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:21 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:21 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:19:21 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:19:22 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:19:22 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:19:22 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.89 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:23 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: systemctl stop couchbase-server.service 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:19:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:19:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.89 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: systemctl start couchbase-server.service 2019-12-05 21:19:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:25 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:19:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:19:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:19:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:19:27 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:28 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: systemctl stop couchbase-server.service 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:19:29 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:19:30 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:30 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: systemctl start couchbase-server.service 2019-12-05 21:19:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:31 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:31 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:19:31 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:19:31 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:19:32 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:19:32 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.15 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:33 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: systemctl stop couchbase-server.service 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:19:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:19:35 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.15 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:36 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: systemctl start couchbase-server.service 2019-12-05 21:19:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:36 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:36 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... Cluster instance shutdown with force 2019-12-05 21:19:46 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2019-12-05 21:19:46 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:19:46 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:19:47 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:19:47 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:19:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:48 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:48 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: systemctl stop couchbase-server.service 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:19:50 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: systemctl start couchbase-server.service 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:19:51 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:19:52 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:19:52 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:19:53 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.102 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:54 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:54 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: systemctl stop couchbase-server.service 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:55 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:19:56 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:19:56 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.102 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:57 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:19:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: systemctl start couchbase-server.service 2019-12-05 21:19:57 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:19:57 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:19:57 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:19:57 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:19:58 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:19:58 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:19:59 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.105 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:19:59 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:19:59 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: systemctl stop couchbase-server.service 2019-12-05 21:20:00 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:00 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:20:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:20:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:20:01 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:01 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:20:01 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.105 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: systemctl start couchbase-server.service 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:20:02 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:20:03 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:20:03 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:20:04 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.107 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:04 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:20:04 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: systemctl stop couchbase-server.service 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:20:06 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:20:07 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.107 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:07 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:20:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: systemctl start couchbase-server.service 2019-12-05 21:20:07 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:07 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:20:07 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:20:08 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:20:09 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:20:09 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:20:10 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.88 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:10 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:20:10 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: systemctl stop couchbase-server.service 2019-12-05 21:20:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:11 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:20:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:20:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:20:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:12 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:20:12 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.88 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: systemctl start couchbase-server.service 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:20:13 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:20:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:20:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:20:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.89 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:15 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:20:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: systemctl stop couchbase-server.service 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:20:17 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.89 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: systemctl start couchbase-server.service 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:20:18 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:20:19 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:20:19 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:20:20 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:20 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:20:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: systemctl stop couchbase-server.service 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:20:22 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: systemctl start couchbase-server.service 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:20:23 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:20:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:20:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:20:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.15 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:26 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:20:26 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: systemctl stop couchbase-server.service 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:20:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:20:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.15 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:20:28 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:20:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: systemctl start couchbase-server.service 2019-12-05 21:20:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:29 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:20:29 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2019-12-05 21:20:39 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.10 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.10', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'43', 'mcdMemoryReserved': 3219, 'storageTotalRam': 4023, 'hostname': u'172.23.105.10:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3781087232, 'memoryTotal': 4219441152, 'memoryQuota': 1748, 'mcdMemoryAllocated': 3219, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:39 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.10:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2156 2019-12-05 21:20:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.105.10:8091:services=kv&password=password&user=Administrator&hostname=172.23.105.10 2019-12-05 21:20:39 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.10:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:20:39 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.10:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.10:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.102 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.102', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'38', 'mcdMemoryReserved': 3171, 'storageTotalRam': 3963, 'hostname': u'172.23.105.102:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3704557568, 'memoryTotal': 4156542976, 'memoryQuota': 1712, 'mcdMemoryAllocated': 3171, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:40 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2124 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.102:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:20:40 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:41 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.102:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:41 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.102:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:41 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:42 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.105 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.105', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'35', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.107.105:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3492147200, 'memoryTotal': 3961196544, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:42 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2019-12-05 21:20:42 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.105:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:20:42 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.105:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.105:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.107.107 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.107.107', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'30', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3778, 'hostname': u'172.23.107.107:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3546484736, 'memoryTotal': 3961716736, 'memoryQuota': 1601, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:43 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.107.107:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.107.107:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:20:43 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.107:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.107:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.97.88 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.88', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'27', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.88:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3500367872, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:44 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.97.88:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.97.88:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:20:44 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:20:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:45 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.88:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:45 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.88:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:45 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:46 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.97.89 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.97.89', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'22', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.97.89:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3515592704, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:46 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.97.89:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2019-12-05 21:20:46 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.97.89:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:46 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:20:46 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.89:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.89:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.98.14 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.14', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'17', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.14:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3506003968, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:47 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.98.14:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.98.14:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:20:47 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.14:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.14:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.98.15 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.98.15', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'12', 'mcdMemoryReserved': 3022, 'storageTotalRam': 3777, 'hostname': u'172.23.98.15:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4926-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3498459136, 'memoryTotal': 3961200640, 'memoryQuota': 1600, 'mcdMemoryAllocated': 3022, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-12-05 21:20:48 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.98.15:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2024 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.98.15:8091:username=Administrator&password=password&port=8091 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:20:48 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:20:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:20:49 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:20:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.15:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:49 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.98.15:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-12-05 21:20:49 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:20:49 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.105.10 **** 2019-12-05 21:20:49 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://172.23.105.10:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2019-12-05 21:20:49 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found." 2019-12-05 21:20:49 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2019-12-05 21:20:54 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2019-12-05 21:20:54 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2019-12-05 21:20:55 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.102:8091 to cluster 2019-12-05 21:20:55 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.102:8091 to this cluster @172.23.105.10:8091 2019-12-05 21:20:58 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.105:8091 to cluster 2019-12-05 21:20:58 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.105:8091 to this cluster @172.23.105.10:8091 2019-12-05 21:21:02 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.107.107:8091 to cluster 2019-12-05 21:21:02 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.107.107:8091 to this cluster @172.23.105.10:8091 2019-12-05 21:21:06 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.102,ns_1@172.23.105.10,ns_1@172.23.107.107,ns_1@172.23.107.105'} 2019-12-05 21:21:06 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-12-05 21:21:06 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-12-05 21:21:06 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-12-05 21:21:16 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-12-05 21:21:16 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0637550354 sec 2019-12-05 21:21:16 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:21:17 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:21:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-12-05 21:21:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:21:17 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.105.10 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_rebalance_in_with_different_topologies ============== 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.102:8091 => {'swap_mem_used': 69206016, 'cpu_utilization': 0, 'mem_free': 3657183232, 'swap_mem_total': 3758092288, 'mem_total': 4156542976, 'services': [u'kv']} 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.10:8091 => {'swap_mem_used': 118001664, 'cpu_utilization': 0, 'mem_free': 3738025984, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'kv']} 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.107:8091 => {'swap_mem_used': 57397248, 'cpu_utilization': 0.7575757575757576, 'mem_free': 3481239552, 'swap_mem_total': 3758092288, 'mem_total': 3961716736, 'services': [u'index', u'n1ql']} 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.105:8091 => {'swap_mem_used': 7634944, 'cpu_utilization': 0.5025125628140703, 'mem_free': 3448213504, 'swap_mem_total': 3758092288, 'mem_total': 3961196544, 'services': [u'eventing']} 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:21:18 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.107.107 port:8091 ssh_username:root] 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root] 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : eventingMemoryQuota=512 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:172.23.107.107 port:8091 ssh_username:root] 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=700 2019-12-05 21:21:31 | INFO | MainProcess | test_thread | [eventing_rebalance.setUp] 100 2019-12-05 21:21:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=100 2019-12-05 21:21:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.02 seconds to create bucket src_bucket 2019-12-05 21:21:32 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:32 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:32 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:33 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2019-12-05 21:21:33 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:33 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:34 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2019-12-05 21:21:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:35 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:35 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2019-12-05 21:21:35 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:36 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:37 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2019-12-05 21:21:37 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:37 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:38 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2019-12-05 21:21:38 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:38 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:21:39 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:21:39 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2019-12-05 21:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=100 2019-12-05 21:21:40 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.03 seconds to create bucket dst_bucket 2019-12-05 21:21:40 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:41 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:42 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2019-12-05 21:21:42 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:42 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:43 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2019-12-05 21:21:43 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:43 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:44 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:44 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2019-12-05 21:21:44 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:45 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:45 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2019-12-05 21:21:45 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:46 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:47 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2019-12-05 21:21:47 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 172.23.105.10 to accept set ops 2019-12-05 21:21:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:47 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 dst_bucket 2019-12-05 21:21:48 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 dst_bucket 2019-12-05 21:21:48 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2019-12-05 21:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.10:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=0&ramQuotaMB=400 2019-12-05 21:21:49 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.02 seconds to create bucket metadata 2019-12-05 21:21:49 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops 2019-12-05 21:21:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:50 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:51 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2019-12-05 21:21:51 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops 2019-12-05 21:21:51 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:52 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:52 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2019-12-05 21:21:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops 2019-12-05 21:21:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:54 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2019-12-05 21:21:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops 2019-12-05 21:21:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2019-12-05 21:21:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops 2019-12-05 21:21:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:56 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2019-12-05 21:21:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 172.23.105.10 to accept set ops 2019-12-05 21:21:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 metadata 2019-12-05 21:21:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 metadata 2019-12-05 21:21:57 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2019-12-05 21:21:59 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] Deploying the following handler code : Function_66360059_test_rebalance_in_with_different_topologies with {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [], 'metadata_bucket': 'metadata'} 2019-12-05 21:21:59 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { delete dst_bucket[context.docID]; } function timerCallback(context) { dst_bucket[context.docID] = context.random_text; } 2019-12-05 21:22:00 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_66360059_test_rebalance_in_with_different_topologies' in metakv", "warnings": null } } 2019-12-05 21:22:00 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_66360059_test_rebalance_in_with_different_topologies to deployed... ... 2019-12-05 21:22:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_66360059_test_rebalance_in_with_different_topologies to deployed... ... 2019-12-05 21:22:41 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 20 secs. Waiting for Function_66360059_test_rebalance_in_with_different_topologies to deployed... ... 2019-12-05 21:23:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:23:01 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:23:03 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.97.88:8091 to cluster 2019-12-05 21:23:03 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.97.88:8091 to this cluster @172.23.105.10:8091 2019-12-05 21:23:05 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.102,ns_1@172.23.105.10,ns_1@172.23.107.107,ns_1@172.23.107.105,ns_1@172.23.97.88'} 2019-12-05 21:23:05 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-12-05 21:23:05 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-12-05 21:23:05 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-12-05 21:23:15 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 50.00 % 2019-12-05 21:23:15 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 50.00% 2019-12-05 21:23:25 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 68.11 % 2019-12-05 21:23:25 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 68.11% 2019-12-05 21:23:35 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 89.84 % 2019-12-05 21:23:35 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 89.84% 2019-12-05 21:23:45 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-12-05 21:23:45 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 40.3441388607 sec 2019-12-05 21:24:01 | INFO | MainProcess | test_thread | [rest_client.rebalance_reached] rebalance reached >100% in 3.0137989521 seconds 2019-12-05 21:24:01 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:24:01 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:24:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 8044 Expected : 40320 ... 2019-12-05 21:24:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 22185 Expected : 40320 ... 2019-12-05 21:25:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 34291 Expected : 40320 ... 2019-12-05 21:25:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 37921 Expected : 40320 ... 2019-12-05 21:26:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 38484 Expected : 40320 ... 2019-12-05 21:26:31 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39345 Expected : 40320 ... 2019-12-05 21:27:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:27:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:28:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:28:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:29:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:29:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:30:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:30:32 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:31:02 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:31:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:32:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:32:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:33:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:33:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:34:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:34:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:35:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 39406 Expected : 40320 ... 2019-12-05 21:35:33 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Documents in source bucket : 40320 2019-12-05 21:35:33 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Final docs count... Current : 40320 Expected : 40320 2019-12-05 21:35:36 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.107.105 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 9, "agg_messages_sent_to_worker": 38081, "agg_queue_memory": 12, "agg_queue_memory_cap": 106954752, "agg_queue_size": 3, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 23455, "dcp_mutation_sent_to_worker": 23455, "dcp_snapshot": 20494, "dcp_stream_req_counter": 1536, "dcp_streamend": 1024, "dcp_streamreq": 1536, "execution_stats": 2409, "failure_stats": 2409, "latency_stats": 2409, "lcb_exception_stats": 2409, "log_level": 3, "num_processed_events": 23455, "processed_events_size": 15780772, "thr_count": 3, "thr_map": 3, "timer_events": 23952, "timer_responses_received": 23451, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 12, "agg_queue_size": 3, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 23455, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 23455, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 4, "lcb_retry_failure": 0, "messages_parsed": 38069, "num_processed_events": 23455, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 23451, "processed_events_size": 15780772, "timer_create_counter": 23451, "timer_create_failure": 0, "timer_msg_counter": 23952, "timer_responses_sent": 0, "timestamp": { "8705": "2019-12-06T05:35:33Z", "8712": "2019-12-06T05:35:33Z", "8713": "2019-12-06T05:35:33Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "8705": "2019-12-06T05:35:33Z", "8712": "2019-12-06T05:35:33Z", "8713": "2019-12-06T05:35:33Z" }, "v8worker_events_lost": 0 }, "function_id": 2517567955, "function_name": "Function_66360059_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "latency_percentile_stats": { "100": 30000, "50": 1800, "80": 2500, "90": 3100, "95": 4000, "99": 8900 }, "lcb_creds_request_counter": 46, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.107.105:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.97.88:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.107.105:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "172.23.97.88:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" } }, "worker_pids": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": 8705, "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": 8712, "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": 8713 } } ] 2019-12-05 21:35:38 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.97.88 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 19, "agg_messages_sent_to_worker": 28377, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 16869, "dcp_mutation_sent_to_worker": 16869, "dcp_snapshot": 12806, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 2195, "failure_stats": 2195, "latency_stats": 2195, "lcb_exception_stats": 2195, "log_level": 3, "num_processed_events": 16869, "processed_events_size": 11356692, "thr_count": 3, "thr_map": 3, "timer_events": 17303, "timer_responses_received": 16869, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 16869, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 16869, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 28365, "num_processed_events": 16869, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 16869, "processed_events_size": 11356692, "timer_create_counter": 16869, "timer_create_failure": 0, "timer_msg_counter": 17303, "timer_responses_sent": 0, "timestamp": { "27439": "2019-12-06T05:35:36Z", "27440": "2019-12-06T05:35:36Z", "27441": "2019-12-06T05:35:36Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27439": "2019-12-06T05:35:36Z", "27440": "2019-12-06T05:35:36Z", "27441": "2019-12-06T05:35:36Z" }, "v8worker_events_lost": 0 }, "function_id": 2517567955, "function_name": "Function_66360059_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" }, "latency_percentile_stats": { "100": 177900, "50": 1800, "80": 4400, "90": 7700, "95": 13400, "99": 64200 }, "lcb_creds_request_counter": 42, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.107.105:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.97.88:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.107.105:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "172.23.97.88:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" } }, "worker_pids": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": 27440, "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": 27441, "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": 27439 } } ] 2019-12-05 21:35:40 | INFO | MainProcess | test_thread | [basetestcase.load] delete 40320 to src_bucket documents... 2019-12-05 21:35:41 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.10:11210 src_bucket 2019-12-05 21:35:41 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.102:11210 src_bucket 2019-12-05 21:36:16 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2019-12-05 21:36:16 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:36:16 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:36:16 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 40320 Expected : 0 ... 2019-12-05 21:36:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 35197 Expected : 0 ... 2019-12-05 21:37:16 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 32747 Expected : 0 ... 2019-12-05 21:37:46 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 19710 Expected : 0 ... 2019-12-05 21:38:16 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 19419 Expected : 0 ... 2019-12-05 21:38:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:39:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:39:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:40:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:40:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:41:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:41:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:42:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:42:47 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:43:17 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:43:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:44:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:44:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:45:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:45:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:46:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:46:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:47:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:47:48 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:48:18 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code Function_66360059_test_rebalance_in_with_different_topologies to complete bucket operations... Current : 18716 Expected : 0 ... 2019-12-05 21:48:49 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Documents in source bucket : 0 2019-12-05 21:48:51 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.107.105 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 11, "agg_messages_sent_to_worker": 70166, "agg_queue_memory": 7124581, "agg_queue_memory_cap": 106954752, "agg_queue_size": 18876, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 20160, "dcp_deletion_sent_to_worker": 20160, "dcp_mutation": 23455, "dcp_mutation_sent_to_worker": 23455, "dcp_snapshot": 40654, "dcp_stream_req_counter": 1536, "dcp_streamend": 1024, "dcp_streamreq": 1536, "execution_stats": 4794, "failure_stats": 4794, "latency_stats": 4794, "lcb_exception_stats": 4794, "log_level": 3, "num_processed_events": 24899, "processed_events_size": 16163760, "thr_count": 3, "thr_map": 3, "timer_events": 25696, "timer_responses_received": 24895, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 7124581, "agg_queue_size": 18876, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 1444, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 23455, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 20160, "enqueued_dcp_mutation_msg_counter": 23455, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 4, "lcb_retry_failure": 0, "messages_parsed": 70154, "num_processed_events": 24899, "on_delete_failure": 0, "on_delete_success": 1444, "on_update_failure": 0, "on_update_success": 23451, "processed_events_size": 16163760, "timer_create_counter": 24895, "timer_create_failure": 0, "timer_msg_counter": 25696, "timer_responses_sent": 0, "timestamp": { "8705": "2019-12-06T05:48:48Z", "8712": "2019-12-06T05:48:48Z", "8713": "2019-12-06T05:48:48Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "8705": "2019-12-06T05:48:48Z", "8712": "2019-12-06T05:48:48Z", "8713": "2019-12-06T05:48:48Z" }, "v8worker_events_lost": 0 }, "function_id": 2517567955, "function_name": "Function_66360059_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "latency_percentile_stats": { "100": 30000, "50": 1800, "80": 2500, "90": 3100, "95": 4000, "99": 8700 }, "lcb_creds_request_counter": 46, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.107.105:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.97.88:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.107.105:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "172.23.97.88:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" } }, "worker_pids": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": 8705, "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": 8712, "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": 8713 } } ] 2019-12-05 21:48:53 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 172.23.97.88 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 31, "agg_messages_sent_to_worker": 60462, "agg_queue_memory": 4, "agg_queue_memory_cap": 106954752, "agg_queue_size": 1, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 20160, "dcp_deletion_sent_to_worker": 20160, "dcp_mutation": 16869, "dcp_mutation_sent_to_worker": 16869, "dcp_snapshot": 32966, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 4580, "failure_stats": 4580, "latency_stats": 4580, "lcb_exception_stats": 4580, "log_level": 3, "num_processed_events": 37029, "processed_events_size": 16716652, "thr_count": 3, "thr_map": 3, "timer_events": 37439, "timer_responses_received": 37029, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 4, "agg_queue_size": 1, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 20160, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 16869, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 20160, "enqueued_dcp_mutation_msg_counter": 16869, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 60450, "num_processed_events": 37029, "on_delete_failure": 0, "on_delete_success": 20160, "on_update_failure": 0, "on_update_success": 16869, "processed_events_size": 16716652, "timer_create_counter": 37029, "timer_create_failure": 0, "timer_msg_counter": 37439, "timer_responses_sent": 0, "timestamp": { "27439": "2019-12-06T05:48:51Z", "27440": "2019-12-06T05:48:51Z", "27441": "2019-12-06T05:48:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27439": "2019-12-06T05:48:51Z", "27440": "2019-12-06T05:48:51Z", "27441": "2019-12-06T05:48:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2517567955, "function_name": "Function_66360059_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" }, "latency_percentile_stats": { "100": 177900, "50": 1300, "80": 2400, "90": 4200, "95": 7200, "99": 35000 }, "lcb_creds_request_counter": 42, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.107.105:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.97.88:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.107.105:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "172.23.97.88:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" } }, "worker_pids": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": 27440, "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": 27441, "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": 27439 } } ] ERROR 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [eventing_base.print_go_routine_dump_from_all_eventing_nodes] Go routine dumps for Node 172.23.107.105 is goroutine profile: total 182 13 @ 0x90f27a 0x91ef90 0xb816ab 0x93cb11 # 0xb816aa net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 9 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa2770c 0xb29894 0xb29372 0xb29653 0xb76c41 0xb76b8d 0xb82dec 0x9e08c2 0x9e05cd 0x9e0018 0xc45bd2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb29893 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb29371 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb29652 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb76c40 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb76b8c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb82deb net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc45bd1 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 9 @ 0x90f27a 0x91ef90 0xb806a3 0x93cb11 # 0xb806a2 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 8 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xce24a0 0xce254a 0xcf6548 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xce249f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0xce2549 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0xcf6547 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf7 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:286 8 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xcf7039 0x93cb11 # 0xcf7038 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 8 @ 0x90f27a 0x90f32e 0x920d3b 0x942ff0 0xce3780 0xce3033 0xce4f5e 0xce5cf0 0x93cb11 # 0x920d3a sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x942fef sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0xce377f gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0xce3032 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0xce4f5d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0xce5cef gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 8 @ 0x90f27a 0x91ef90 0xcf61c2 0x93cb11 # 0xcf61c1 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:257 5 @ 0x90f27a 0x91ef90 0x1067343 0x93cb11 # 0x1067342 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 4 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa26b7a 0xb800e5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb800e4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 4 @ 0x90f27a 0x91ef90 0xcca98f 0x93cb11 # 0xcca98e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 4 @ 0x90f27a 0x91ef90 0xd302f4 0x93cb11 # 0xd302f3 gopkg.in/couchbase/gocb%2ev1.(*ThresholdLoggingTracer).loggerRoutine+0x133 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/thresholdlogtracer.go:244 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e023a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e0239 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:102 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e04e2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e04e1 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:115 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0x965296 0x965408 0xc69679 0xc76c01 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x965295 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x965407 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0xc69678 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0xc76c00 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x220 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1185 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094875 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094874 github.com/couchbase/eventing/consumer.(*Consumer).feedbackReadMessageLoop+0x2b4 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:728 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094f9a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094f99 github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop+0x2a9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:792 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x11142b1 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x11142b0 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:599 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x1114c31 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x1114c30 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:633 3 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xc8aeae 0x93cb11 # 0xc8aead github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:307 3 @ 0x90f27a 0x90f32e 0x91fc14 0x91f839 0x945fa2 0x10bebaa 0x1069903 0x93cb11 # 0x91f838 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x945fa1 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0x10beba9 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1c19 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:292 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 3 @ 0x90f27a 0x91ef90 0x1077b8d 0x93cb11 # 0x1077b8c github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ac /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 3 @ 0x90f27a 0x91ef90 0x107bcec 0x93cb11 # 0x107bceb github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x1db /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x90f27a 0x91ef90 0x1092ade 0x93cb11 # 0x1092add github.com/couchbase/eventing/consumer.(*Consumer).sendMessageLoop+0x18d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:561 3 @ 0x90f27a 0x91ef90 0x1098377 0x93cb11 # 0x1098376 github.com/couchbase/eventing/consumer.(*Consumer).processDCPEvents+0x736 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:57 3 @ 0x90f27a 0x91ef90 0x109d9f8 0x93cb11 # 0x109d9f7 github.com/couchbase/eventing/consumer.(*Consumer).processStatsEvents+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:345 3 @ 0x90f27a 0x91ef90 0x109e883 0x93cb11 # 0x109e882 github.com/couchbase/eventing/consumer.(*Consumer).processFilterEvents+0x2e2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:395 3 @ 0x90f27a 0x91ef90 0x10a91fa 0x93cb11 # 0x10a91f9 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x149 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:972 3 @ 0x90f27a 0x91ef90 0x10ad2be 0x93cb11 # 0x10ad2bd github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1197 3 @ 0x90f27a 0x91ef90 0x10b8f75 0x93cb11 # 0x10b8f74 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0x114 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:102 3 @ 0x90f27a 0x91ef90 0x10e50a0 0x93cb11 # 0x10e509f github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:639 3 @ 0x90f27a 0x91ef90 0x1114485 0x93cb11 # 0x1114484 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:603 3 @ 0x90f27a 0x91ef90 0x1114e05 0x93cb11 # 0x1114e04 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:637 3 @ 0x90f27a 0x91ef90 0xc6de6b 0x93cb11 # 0xc6de6a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:190 3 @ 0x950c45 0x9b1358 0x9aad3c 0x9aa2eb 0xfb298c 0x107a964 0x1069903 0x93cb11 # 0x950c44 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x9b1357 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x9aad3b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x9aa2ea os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xfb298b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0x107a963 github.com/couchbase/eventing/consumer.(*client).Serve+0x1113 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:125 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 2 @ 0x90f27a 0x91ef90 0x104b591 0x93cb11 # 0x104b590 github.com/couchbase/eventing/util.(*Stats).updater+0x100 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/stats.go:54 2 @ 0x90f27a 0x91ef90 0x11168fc 0x93cb11 # 0x11168fb github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x8f1e12 0x923d26 0xd4d312 0x93cb11 # 0x923d25 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd4d311 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0x111678d 0x114c36e 0x11687c0 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x111678c github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x114c36d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:445 # 0x11687bf github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0xc3f340 0xc409f5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc3f33f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc409f4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb6784a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67849 net/http.(*connReader).backgroundRead+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:668 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb67cf8 0xa26a2e 0xa26b7a 0xb6cf83 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67cf7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb6cf82 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xaecf77 0xb70ed5 0xb714eb 0xb71dde 0x1168f26 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaecf76 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb714ea net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb71ddd net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x1168f25 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:228 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xb70ed5 0xb70c29 0x11689a7 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb70c28 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x11689a6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:186 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x117ce25 0x118820c 0x90ee22 0x93cb11 # 0x117ce24 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x118820b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x90ee21 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x1183093 0x93cb11 # 0x1183092 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:79 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69eb 0x10434c7 0x93cb11 # 0x10434c6 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x1044c23 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x1044c22 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x10fba14 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x10fba13 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:142 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x11066dc 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x11066db github.com/couchbase/eventing/producer.(*Producer).updateStats+0x14b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:873 1 @ 0x90f27a 0x91ef90 0x10fe7e1 0x1069903 0x93cb11 # 0x10fe7e0 github.com/couchbase/eventing/producer.(*Producer).Serve+0x17a0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:198 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x90f27a 0x91ef90 0x1106df5 0x93cb11 # 0x1106df4 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:897 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1151cf2 0x1116bc4 0x1115feb 0x11163e3 0x1117582 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1151cf1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1116bc3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11163e2 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1152482 0x1116a14 0x1115feb 0x1116283 0x11174b2 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1152481 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1116a13 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1116282 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0xb98e91 0x93cb11 # 0xb98e90 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x90f27a 0x91ef90 0xb992ff 0x93cb11 # 0xb992fe github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116924a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169249 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:242 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x11693ca 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x11693c9 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:253 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116954a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169549 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:264 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188939 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188938 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188a49 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188a48 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188b59 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188b58 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188c69 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188c68 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188d79 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188d78 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188e89 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188e88 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0xd44938 0xd44740 0xd41284 0xd4ca4d 0xd4cdd1 0x11463ad 0x1169fb8 0xb6ddf4 0xb6fa60 0xb70b0c 0xb6ce11 0x93cb11 # 0xd44937 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd4473f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd41283 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd4ca4c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd4cdd0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11463ac github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3578 # 0x1169fb7 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb6ddf3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb6fa5f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb70b0b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb6ce10 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [eventing_base.print_go_routine_dump_from_all_eventing_nodes] Go routine dumps for Node 172.23.97.88 is goroutine profile: total 194 13 @ 0x90f27a 0x91ef90 0xb816ab 0x93cb11 # 0xb816aa net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 9 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa2770c 0xb29894 0xb29372 0xb29653 0xb76c41 0xb76b8d 0xb82dec 0x9e08c2 0x9e05cd 0x9e0018 0xc45bd2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb29893 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb29371 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb29652 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb76c40 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb76b8c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb82deb net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc45bd1 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 9 @ 0x90f27a 0x91ef90 0xb806a3 0x93cb11 # 0xb806a2 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 8 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xce24a0 0xce254a 0xcf6548 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xce249f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0xce2549 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0xcf6547 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf7 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:286 8 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xcf7039 0x93cb11 # 0xcf7038 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 8 @ 0x90f27a 0x90f32e 0x920d3b 0x942ff0 0xce3780 0xce3033 0xce4f5e 0xce5cf0 0x93cb11 # 0x920d3a sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x942fef sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0xce377f gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0xce3032 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0xce4f5d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0xce5cef gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 8 @ 0x90f27a 0x91ef90 0xcf61c2 0x93cb11 # 0xcf61c1 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:257 6 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0x965296 0x965408 0xc69679 0xc76c01 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x965295 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x965407 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0xc69678 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0xc76c00 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x220 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1185 6 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xc8aeae 0x93cb11 # 0xc8aead github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:307 6 @ 0x90f27a 0x91ef90 0x10e50a0 0x93cb11 # 0x10e509f github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:639 6 @ 0x90f27a 0x91ef90 0xc6de6b 0x93cb11 # 0xc6de6a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:190 5 @ 0x90f27a 0x91ef90 0x1067343 0x93cb11 # 0x1067342 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 4 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa26b7a 0xb800e5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb800e4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 4 @ 0x90f27a 0x91ef90 0xcca98f 0x93cb11 # 0xcca98e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 4 @ 0x90f27a 0x91ef90 0xd302f4 0x93cb11 # 0xd302f3 gopkg.in/couchbase/gocb%2ev1.(*ThresholdLoggingTracer).loggerRoutine+0x133 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/thresholdlogtracer.go:244 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e023a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e0239 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:102 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e04e2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e04e1 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:115 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094875 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094874 github.com/couchbase/eventing/consumer.(*Consumer).feedbackReadMessageLoop+0x2b4 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:728 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094f9a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094f99 github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop+0x2a9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:792 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x11142b1 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x11142b0 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:599 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x1114c31 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x1114c30 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:633 3 @ 0x90f27a 0x90f32e 0x91fc14 0x91f839 0x945fa2 0x10bebaa 0x1069903 0x93cb11 # 0x91f838 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x945fa1 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0x10beba9 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1c19 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:292 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 3 @ 0x90f27a 0x91ef90 0x107bcec 0x93cb11 # 0x107bceb github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x1db /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x90f27a 0x91ef90 0x1092ade 0x93cb11 # 0x1092add github.com/couchbase/eventing/consumer.(*Consumer).sendMessageLoop+0x18d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:561 3 @ 0x90f27a 0x91ef90 0x1098377 0x93cb11 # 0x1098376 github.com/couchbase/eventing/consumer.(*Consumer).processDCPEvents+0x736 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:57 3 @ 0x90f27a 0x91ef90 0x109d9f8 0x93cb11 # 0x109d9f7 github.com/couchbase/eventing/consumer.(*Consumer).processStatsEvents+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:345 3 @ 0x90f27a 0x91ef90 0x109e883 0x93cb11 # 0x109e882 github.com/couchbase/eventing/consumer.(*Consumer).processFilterEvents+0x2e2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:395 3 @ 0x90f27a 0x91ef90 0x10a91fa 0x93cb11 # 0x10a91f9 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x149 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:972 3 @ 0x90f27a 0x91ef90 0x10ad2be 0x93cb11 # 0x10ad2bd github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1197 3 @ 0x90f27a 0x91ef90 0x10b8f75 0x93cb11 # 0x10b8f74 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0x114 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:102 3 @ 0x90f27a 0x91ef90 0x1114485 0x93cb11 # 0x1114484 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:603 3 @ 0x90f27a 0x91ef90 0x1114e05 0x93cb11 # 0x1114e04 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:637 3 @ 0x950c45 0x9b1358 0x9aad3c 0x9aa2eb 0xfb298c 0x107a964 0x1069903 0x93cb11 # 0x950c44 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x9b1357 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x9aad3b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x9aa2ea os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xfb298b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0x107a963 github.com/couchbase/eventing/consumer.(*client).Serve+0x1113 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:125 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 2 @ 0x90f27a 0x91ef90 0x104b591 0x93cb11 # 0x104b590 github.com/couchbase/eventing/util.(*Stats).updater+0x100 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/stats.go:54 2 @ 0x90f27a 0x91ef90 0x1077b8d 0x93cb11 # 0x1077b8c github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ac /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 2 @ 0x90f27a 0x91ef90 0x11168fc 0x93cb11 # 0x11168fb github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x8f1e12 0x923d26 0xd4d312 0x93cb11 # 0x923d25 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd4d311 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0x111678d 0x114c36e 0x11687c0 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x111678c github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x114c36d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:445 # 0x11687bf github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0xc3f340 0xc409f5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc3f33f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc409f4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb67cf8 0xa26a2e 0xa26b7a 0xb6cf83 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67cf7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb6cf82 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xaecf77 0xb70ed5 0xb714eb 0xb71dde 0x1168f26 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaecf76 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb714ea net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb71ddd net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x1168f25 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:228 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xb70ed5 0xb70c29 0x11689a7 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb70c28 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x11689a6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:186 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x117ce25 0x118820c 0x90ee22 0x93cb11 # 0x117ce24 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x118820b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x90ee21 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x1183093 0x93cb11 # 0x1183092 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:79 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69eb 0x10434c7 0x93cb11 # 0x10434c6 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x1044c23 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x1044c22 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x10fba14 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x10fba13 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:142 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x11066dc 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x11066db github.com/couchbase/eventing/producer.(*Producer).updateStats+0x14b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:873 1 @ 0x90f27a 0x91ef90 0x10fe7e1 0x1069903 0x93cb11 # 0x10fe7e0 github.com/couchbase/eventing/producer.(*Producer).Serve+0x17a0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:198 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x90f27a 0x91ef90 0x1106df5 0x93cb11 # 0x1106df4 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:897 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1151cf2 0x1116bc4 0x1115feb 0x11163e3 0x1117582 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1151cf1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1116bc3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11163e2 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1152482 0x1116a14 0x1115feb 0x1116283 0x11174b2 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1152481 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1116a13 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1116282 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0xb98e91 0x93cb11 # 0xb98e90 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x90f27a 0x91ef90 0xb992ff 0x93cb11 # 0xb992fe github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116924a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169249 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:242 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x11693ca 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x11693c9 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:253 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116954a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169549 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:264 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188939 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188938 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188a49 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188a48 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188b59 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188b58 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188c69 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188c68 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188d79 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188d78 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188e89 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188e88 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0x90f27a 0x91ef90 0xd037b3 0xd03f5f 0xd02426 0x10d0499 0x104947a 0x10768ef 0x93cb11 # 0xd037b2 gopkg.in/couchbase/gocb%2ev1.(*opManager).Wait+0x112 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/bucket_crud.go:163 # 0xd03f5e gopkg.in/couchbase/gocb%2ev1.(*Bucket).get+0x23e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/bucket_crud.go:210 # 0xd02425 gopkg.in/couchbase/gocb%2ev1.(*Bucket).Get+0xe5 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/bucket_crud.go:13 # 0x10d0498 github.com/couchbase/eventing/consumer.glob..func5+0x428 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/bucket_ops.go:193 # 0x1049479 github.com/couchbase/eventing/util.Retry+0x139 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry.go:65 # 0x10768ee github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x70e /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:58 1 @ 0xb677f0 0x93cb11 # 0xb677f0 net/http.(*connReader).backgroundRead+0x0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:667 1 @ 0xd44938 0xd44740 0xd41284 0xd4ca4d 0xd4cdd1 0x11463ad 0x1169fb8 0xb6ddf4 0xb6fa60 0xb70b0c 0xb6ce11 0x93cb11 # 0xd44937 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd4473f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd41283 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd4ca4c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd4cdd0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11463ac github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3578 # 0x1169fb7 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb6ddf3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb6fa5f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb70b0b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb6ce10 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.107.105 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 11, "agg_messages_sent_to_worker": 70241, "agg_queue_memory": 7124589, "agg_queue_memory_cap": 106954752, "agg_queue_size": 18878, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 20160, "dcp_deletion_sent_to_worker": 20160, "dcp_mutation": 23455, "dcp_mutation_sent_to_worker": 23455, "dcp_snapshot": 40654, "dcp_stream_req_counter": 1536, "dcp_streamend": 1024, "dcp_streamreq": 1536, "execution_stats": 4809, "failure_stats": 4809, "latency_stats": 4809, "lcb_exception_stats": 4809, "log_level": 3, "num_processed_events": 24899, "processed_events_size": 16163760, "thr_count": 3, "thr_map": 3, "timer_events": 25696, "timer_responses_received": 24895, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 7124589, "agg_queue_size": 18878, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 1444, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 23455, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 20160, "enqueued_dcp_mutation_msg_counter": 23455, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 4, "lcb_retry_failure": 0, "messages_parsed": 70229, "num_processed_events": 24899, "on_delete_failure": 0, "on_delete_success": 1444, "on_update_failure": 0, "on_update_success": 23451, "processed_events_size": 16163760, "timer_create_counter": 24895, "timer_create_failure": 0, "timer_msg_counter": 25696, "timer_responses_sent": 0, "timestamp": { "8705": "2019-12-06T05:48:53Z", "8712": "2019-12-06T05:48:53Z", "8713": "2019-12-06T05:48:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "8705": "2019-12-06T05:48:53Z", "8712": "2019-12-06T05:48:53Z", "8713": "2019-12-06T05:48:53Z" }, "v8worker_events_lost": 0 }, "function_id": 2517567955, "function_name": "Function_66360059_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "latency_percentile_stats": { "100": 30000, "50": 1800, "80": 2500, "90": 3100, "95": 4000, "99": 8700 }, "lcb_creds_request_counter": 46, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.107.105:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.97.88:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.107.105:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "172.23.97.88:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" } }, "worker_pids": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": 8705, "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": 8712, "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": 8713 } } ] 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [eventing_base.print_eventing_stats_from_all_eventing_nodes] Stats for Node 172.23.97.88 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 31, "agg_messages_sent_to_worker": 60507, "agg_queue_memory": 4, "agg_queue_memory_cap": 106954752, "agg_queue_size": 1, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 20160, "dcp_deletion_sent_to_worker": 20160, "dcp_mutation": 16869, "dcp_mutation_sent_to_worker": 16869, "dcp_snapshot": 32966, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 4589, "failure_stats": 4589, "latency_stats": 4589, "lcb_exception_stats": 4589, "log_level": 3, "num_processed_events": 37029, "processed_events_size": 16716652, "thr_count": 3, "thr_map": 3, "timer_events": 37439, "timer_responses_received": 37029, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 4, "agg_queue_size": 1, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 20160, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 16869, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 20160, "enqueued_dcp_mutation_msg_counter": 16869, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 60495, "num_processed_events": 37029, "on_delete_failure": 0, "on_delete_success": 20160, "on_update_failure": 0, "on_update_success": 16869, "processed_events_size": 16716652, "timer_create_counter": 37029, "timer_create_failure": 0, "timer_msg_counter": 37439, "timer_responses_sent": 0, "timestamp": { "27439": "2019-12-06T05:48:54Z", "27440": "2019-12-06T05:48:54Z", "27441": "2019-12-06T05:48:54Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "27439": "2019-12-06T05:48:54Z", "27440": "2019-12-06T05:48:54Z", "27441": "2019-12-06T05:48:54Z" }, "v8worker_events_lost": 0 }, "function_id": 2517567955, "function_name": "Function_66360059_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" }, "latency_percentile_stats": { "100": 177900, "50": 1300, "80": 2400, "90": 4200, "95": 7200, "99": 35000 }, "lcb_creds_request_counter": 42, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.107.105:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.97.88:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.107.105:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[0-170]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[171-341]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[342-511]" }, "172.23.97.88:8096": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": "[512-682]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": "[683-853]", "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": "[854-1023]" } }, "worker_pids": { "worker_Function_66360059_test_rebalance_in_with_different_topologies_0": 27440, "worker_Function_66360059_test_rebalance_in_with_different_topologies_1": 27441, "worker_Function_66360059_test_rebalance_in_with_different_topologies_2": 27439 } } ] 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:172.23.107.105 port:8091 ssh_username:root, ip:172.23.97.88 port:8091 ssh_username:root] 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:48:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:48:55 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.107.105:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-12-05 21:48:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2019-12-05 21:48:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:48:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2019-12-05 21:48:55 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:48:55 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:48:56 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:48:56 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 172.23.97.88:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2019-12-05 21:48:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2019-12-05 21:48:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:48:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2019-12-05 21:48:56 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [eventing_base.tearDown] Bucket dst_bucket DGM is 100 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [eventing_base.tearDown] Bucket metadata DGM is 100 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [eventing_base.tearDown] Bucket src_bucket DGM is 100 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.102:8091 => {'swap_mem_used': 69206016, 'cpu_utilization': 8.07799442896936, 'mem_free': 3383386112, 'swap_mem_total': 3758092288, 'mem_total': 4156542976, 'services': [u'kv']} 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.10:8091 => {'swap_mem_used': 118001664, 'cpu_utilization': 16.71309192200557, 'mem_free': 3470213120, 'swap_mem_total': 3758092288, 'mem_total': 4219441152, 'services': [u'kv']} 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.107:8091 => {'swap_mem_used': 57397248, 'cpu_utilization': 0.5037783375314862, 'mem_free': 3385647104, 'swap_mem_total': 3758092288, 'mem_total': 3961716736, 'services': [u'index', u'n1ql']} 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.107.105:8091 => {'swap_mem_used': 7634944, 'cpu_utilization': 11.91860465116279, 'mem_free': 3181252608, 'swap_mem_total': 3758092288, 'mem_total': 3961196544, 'services': [u'eventing']} 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.97.88:8091 => {'swap_mem_used': 0, 'cpu_utilization': 11.38888888888889, 'mem_free': 3202179072, 'swap_mem_total': 3758092288, 'mem_total': 3961200640, 'services': [u'eventing']} 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:48:57 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:48:59 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:48:59 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:49:00 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:49:00 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:49:01 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:49:01 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:49:02 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:49:02 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:49:03 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:49:03 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:49:04 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:49:04 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:49:05 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:49:05 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:49:06 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:49:07 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:49:07 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:49:08 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:49:09 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:09 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:09 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: systemctl stop couchbase-server.service 2019-12-05 21:49:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:11 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:11 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:12 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:12 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.10 with username:root 2019-12-05 21:49:12 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.10 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.10 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.10: systemctl start couchbase-server.service 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:49:13 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:49:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:49:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:49:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.102 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:16 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:16 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: systemctl stop couchbase-server.service 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.102 with username:root 2019-12-05 21:49:18 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.102 2019-12-05 21:49:19 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.105.102 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:20 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.102: systemctl start couchbase-server.service 2019-12-05 21:49:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:20 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:20 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:49:20 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:49:21 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:49:21 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:49:22 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.105 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:22 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: systemctl stop couchbase-server.service 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.105 with username:root 2019-12-05 21:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.105 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.105 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.105: systemctl start couchbase-server.service 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:49:27 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.107 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:27 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: systemctl stop couchbase-server.service 2019-12-05 21:49:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:29 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:30 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:30 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.107.107 with username:root 2019-12-05 21:49:30 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.107.107 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.107.107 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.107.107: systemctl start couchbase-server.service 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:49:31 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:49:32 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:49:32 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:49:33 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.88 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:33 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:33 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: systemctl stop couchbase-server.service 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.88 with username:root 2019-12-05 21:49:35 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.88 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.88 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.88: systemctl start couchbase-server.service 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:49:36 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:49:37 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:49:37 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:49:38 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.89 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:38 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: systemctl stop couchbase-server.service 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.97.89 with username:root 2019-12-05 21:49:40 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.97.89 2019-12-05 21:49:41 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.97.89 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:41 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.97.89: systemctl start couchbase-server.service 2019-12-05 21:49:41 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:41 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:41 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:49:42 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:49:42 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:49:42 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:49:43 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:43 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: systemctl stop couchbase-server.service 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.14 with username:root 2019-12-05 21:49:45 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.14 2019-12-05 21:49:46 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.14 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:46 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.14: systemctl start couchbase-server.service 2019-12-05 21:49:46 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:46 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:47 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:49:47 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:49:47 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:49:48 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:49:48 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.15 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:49 | INFO | MainProcess | test_thread | [remote_util.stop_couchbase] Running systemd command on this server 2019-12-05 21:49:49 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: systemctl stop couchbase-server.service 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.98.15 with username:root 2019-12-05 21:49:50 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.98.15 2019-12-05 21:49:51 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 172.23.98.15 **** The version file /opt/couchbase/ VERSION.txt exists 2019-12-05 21:49:52 | INFO | MainProcess | test_thread | [remote_util.start_couchbase] Running systemd command on this server 2019-12-05 21:49:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.98.15: systemctl start couchbase-server.service 2019-12-05 21:49:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-12-05 21:49:52 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2019-12-05 21:49:52 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... ====================================================================== ERROR: test_rebalance_in_with_different_topologies (eventing.eventing_rebalance.EventingRebalance) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/eventing/eventing_rebalance.py", line 445, in test_rebalance_in_with_different_topologies self.verify_eventing_results(self.function_name, 0, skip_stats_validation=True) File "./pytests/eventing/eventing_base.py", line 272, in verify_eventing_results timers_in_past,lcb)) Exception: Bucket operations from handler code took lot of time to complete or didn't go through. Current : 18716 Expected : 0 dcp_backlog : 0 TIMERS_IN_PAST : 0 lcb_exceptions : {'172.23.107.105': {}, '172.23.97.88': {}} ---------------------------------------------------------------------- Ran 1 test in 1869.726s FAILED (errors=1)