[2021-03-25 08:42:17,076] - [eventing_base] [140232468973312] - INFO - Starting Test: test_vb_shuffle_during_failover_add_back [2021-03-25 08:42:17,080] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:42:17,214] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.148 [2021-03-25 08:42:17,548] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:17,984] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:18,101] - [rest_client] [140232468973312] - INFO - Node version in cluster 6.6.2-9577-enterprise [2021-03-25 08:42:18,130] - [rest_client] [140232468973312] - ERROR - GET http://172.23.123.148:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:42:18,131] - [rest_client] [140232468973312] - INFO - Node versions in cluster [] [2021-03-25 08:42:18,131] - [basetestcase] [140232468973312] - INFO - ============== basetestcase setup was started for test #2 test_vb_shuffle_during_failover_add_back============== [2021-03-25 08:42:18,140] - [rest_client] [140232468973312] - ERROR - GET http://172.23.123.148:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:42:18,357] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.123.148:8091 [2021-03-25 08:42:18,363] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.123.148:8091 is running [2021-03-25 08:42:18,369] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.123.150:8091 [2021-03-25 08:42:18,375] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.123.150:8091 is running [2021-03-25 08:42:18,383] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.105.224:8091 [2021-03-25 08:42:18,390] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.105.224:8091 is running [2021-03-25 08:42:18,396] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.121.217:8091 [2021-03-25 08:42:18,401] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.121.217:8091 is running [2021-03-25 08:42:18,407] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.121.219:8091 [2021-03-25 08:42:18,412] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.121.219:8091 is running [2021-03-25 08:42:18,419] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.121.210:8091 [2021-03-25 08:42:18,425] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.121.210:8091 is running [2021-03-25 08:42:18,431] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.121.216:8091 [2021-03-25 08:42:18,436] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.121.216:8091 is running [2021-03-25 08:42:18,444] - [cluster_helper] [140232468973312] - INFO - waiting for ns_server @ 172.23.121.70:8091 [2021-03-25 08:42:18,450] - [cluster_helper] [140232468973312] - INFO - ns_server @ 172.23.121.70:8091 is running [2021-03-25 08:42:18,451] - [basetestcase] [140232468973312] - INFO - initializing cluster [2021-03-25 08:42:18,454] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:42:18,587] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.148 [2021-03-25 08:42:18,886] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:19,342] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:19,356] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:42:19,489] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.148 [2021-03-25 08:42:19,789] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:20,206] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:20,295] - [remote_util] [140232468973312] - INFO - 172.23.123.148 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:20,433] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:20,433] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.148: systemctl stop couchbase-server.service [2021-03-25 08:42:22,004] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:22,005] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:42:22,119] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.148: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:42:22,141] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:22,142] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.148: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:42:22,224] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:22,227] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:42:22,361] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.148 [2021-03-25 08:42:22,700] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:23,116] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:23,203] - [remote_util] [140232468973312] - INFO - 172.23.123.148 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:23,203] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:23,313] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:42:23,448] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:23,448] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.148: systemctl start couchbase-server.service [2021-03-25 08:42:23,484] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:23,484] - [remote_util] [140232468973312] - INFO - 172.23.123.148:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:42:28,490] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.148: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:42:28,528] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:28,528] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 110097 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:42:28,528] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:28,632] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:42:28,749] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.150 with username:root [2021-03-25 08:42:28,883] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.150 [2021-03-25 08:42:29,158] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:29,613] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:29,628] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.150 with username:root [2021-03-25 08:42:29,762] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.150 [2021-03-25 08:42:30,061] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:30,515] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:30,608] - [remote_util] [140232468973312] - INFO - 172.23.123.150 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:30,689] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:30,689] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.150: systemctl stop couchbase-server.service [2021-03-25 08:42:32,260] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:32,261] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:42:32,325] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.150: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:42:32,345] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:32,346] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.150: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:42:32,429] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:32,431] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.150 with username:root [2021-03-25 08:42:32,565] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.150 [2021-03-25 08:42:32,839] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:33,294] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:33,383] - [remote_util] [140232468973312] - INFO - 172.23.123.150 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:33,383] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:33,492] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:42:33,694] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:33,694] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.150: systemctl start couchbase-server.service [2021-03-25 08:42:33,729] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:33,730] - [remote_util] [140232468973312] - INFO - 172.23.123.150:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:42:38,735] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.150: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:42:38,783] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:38,783] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 90789 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:42:38,783] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:38,890] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:42:39,007] - [remote_util] [140232468973312] - INFO - connecting to 172.23.105.224 with username:root [2021-03-25 08:42:39,210] - [remote_util] [140232468973312] - INFO - Connected to 172.23.105.224 [2021-03-25 08:42:40,030] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:40,944] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:40,973] - [remote_util] [140232468973312] - INFO - connecting to 172.23.105.224 with username:root [2021-03-25 08:42:41,141] - [remote_util] [140232468973312] - INFO - Connected to 172.23.105.224 [2021-03-25 08:42:41,950] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:42,832] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:42,947] - [remote_util] [140232468973312] - INFO - 172.23.105.224 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:43,301] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:43,302] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: systemctl stop couchbase-server.service [2021-03-25 08:42:45,433] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:45,433] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:42:45,547] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:42:45,668] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:45,668] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:42:45,788] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:45,791] - [remote_util] [140232468973312] - INFO - connecting to 172.23.105.224 with username:root [2021-03-25 08:42:45,957] - [remote_util] [140232468973312] - INFO - Connected to 172.23.105.224 [2021-03-25 08:42:46,692] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:47,489] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:47,586] - [remote_util] [140232468973312] - INFO - 172.23.105.224 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:47,586] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:47,709] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:42:48,030] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:48,030] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: systemctl start couchbase-server.service [2021-03-25 08:42:48,152] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:48,152] - [remote_util] [140232468973312] - INFO - 172.23.105.224:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:42:53,157] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:42:53,281] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:53,281] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 21471 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:42:53,281] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:53,434] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:42:53,550] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.217 with username:root [2021-03-25 08:42:53,649] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.217 [2021-03-25 08:42:53,951] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:54,377] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:54,392] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.217 with username:root [2021-03-25 08:42:54,489] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.217 [2021-03-25 08:42:54,752] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:55,179] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:55,263] - [remote_util] [140232468973312] - INFO - 172.23.121.217 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:55,392] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:55,392] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: systemctl stop couchbase-server.service [2021-03-25 08:42:56,913] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:56,913] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:42:56,977] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:42:56,997] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:56,997] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:42:57,080] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:57,083] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.217 with username:root [2021-03-25 08:42:57,180] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.217 [2021-03-25 08:42:57,469] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:42:57,892] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:42:57,976] - [remote_util] [140232468973312] - INFO - 172.23.121.217 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:42:57,977] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:42:58,080] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:42:58,212] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:42:58,212] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: systemctl start couchbase-server.service [2021-03-25 08:42:58,248] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:42:58,248] - [remote_util] [140232468973312] - INFO - 172.23.121.217:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:43:03,253] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:43:03,290] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:03,291] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 104085 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:43:03,291] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:03,392] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:43:03,509] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.219 with username:root [2021-03-25 08:43:03,644] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.219 [2021-03-25 08:43:03,935] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:04,362] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:04,379] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.219 with username:root [2021-03-25 08:43:04,477] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.219 [2021-03-25 08:43:04,740] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:05,169] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:05,257] - [remote_util] [140232468973312] - INFO - 172.23.121.219 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:05,386] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:05,386] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.219: systemctl stop couchbase-server.service [2021-03-25 08:43:06,907] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:06,908] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:43:06,972] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.219: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:43:06,995] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:06,995] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.219: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:43:07,077] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:07,080] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.219 with username:root [2021-03-25 08:43:07,214] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.219 [2021-03-25 08:43:07,508] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:07,941] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:08,028] - [remote_util] [140232468973312] - INFO - 172.23.121.219 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:08,028] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:08,132] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:43:08,267] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:08,267] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.219: systemctl start couchbase-server.service [2021-03-25 08:43:08,303] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:08,303] - [remote_util] [140232468973312] - INFO - 172.23.121.219:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:43:13,308] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.219: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:43:13,346] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:13,346] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 104236 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:43:13,346] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:13,449] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:43:13,566] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.210 with username:root [2021-03-25 08:43:13,700] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.210 [2021-03-25 08:43:13,970] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:14,396] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:14,410] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.210 with username:root [2021-03-25 08:43:14,508] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.210 [2021-03-25 08:43:14,811] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:15,240] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:15,325] - [remote_util] [140232468973312] - INFO - 172.23.121.210 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:15,395] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:15,395] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.210: systemctl stop couchbase-server.service [2021-03-25 08:43:16,966] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:16,967] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:43:17,031] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.210: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:43:17,051] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:17,052] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.210: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:43:17,134] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:17,137] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.210 with username:root [2021-03-25 08:43:17,270] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.210 [2021-03-25 08:43:17,529] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:17,960] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:18,046] - [remote_util] [140232468973312] - INFO - 172.23.121.210 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:18,046] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:18,151] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:43:18,284] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:18,284] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.210: systemctl start couchbase-server.service [2021-03-25 08:43:18,319] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:18,320] - [remote_util] [140232468973312] - INFO - 172.23.121.210:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:43:23,325] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.210: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:43:23,362] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:23,362] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 118908 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:43:23,363] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:23,431] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:43:23,547] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.216 with username:root [2021-03-25 08:43:23,682] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.216 [2021-03-25 08:43:23,947] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:24,375] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:24,388] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.216 with username:root [2021-03-25 08:43:24,485] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.216 [2021-03-25 08:43:24,776] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:25,209] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:25,299] - [remote_util] [140232468973312] - INFO - 172.23.121.216 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:25,496] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:25,497] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.216: systemctl stop couchbase-server.service [2021-03-25 08:43:27,018] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:27,018] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:43:27,132] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.216: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:43:27,153] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:27,153] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.216: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:43:27,236] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:27,238] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.216 with username:root [2021-03-25 08:43:27,372] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.216 [2021-03-25 08:43:27,637] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:28,064] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:28,151] - [remote_util] [140232468973312] - INFO - 172.23.121.216 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:28,151] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:28,256] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:43:28,387] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:28,387] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.216: systemctl start couchbase-server.service [2021-03-25 08:43:28,424] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:28,424] - [remote_util] [140232468973312] - INFO - 172.23.121.216:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:43:33,429] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.216: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:43:33,466] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:33,466] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 58407 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:43:33,466] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:33,566] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:43:33,684] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.70 with username:root [2021-03-25 08:43:33,818] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.70 [2021-03-25 08:43:34,087] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:34,539] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:34,553] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.70 with username:root [2021-03-25 08:43:34,650] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.70 [2021-03-25 08:43:34,979] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:35,413] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:35,500] - [remote_util] [140232468973312] - INFO - 172.23.121.70 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:35,569] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:35,569] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.70: systemctl stop couchbase-server.service [2021-03-25 08:43:37,090] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:37,090] - [basetestcase] [140232468973312] - INFO - Couchbase stopped [2021-03-25 08:43:37,205] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.70: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-03-25 08:43:37,225] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:37,225] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.70: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-03-25 08:43:37,308] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:37,310] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.70 with username:root [2021-03-25 08:43:37,444] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.70 [2021-03-25 08:43:37,713] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:38,142] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:38,228] - [remote_util] [140232468973312] - INFO - 172.23.121.70 **** The version file /opt/couchbase/ VERSION.txt exists [2021-03-25 08:43:38,228] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:38,344] - [remote_util] [140232468973312] - INFO - Starting couchbase server [2021-03-25 08:43:38,476] - [remote_util] [140232468973312] - INFO - Running systemd command on this server [2021-03-25 08:43:38,476] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.70: systemctl start couchbase-server.service [2021-03-25 08:43:38,513] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:38,513] - [remote_util] [140232468973312] - INFO - 172.23.121.70:sleep for 5 secs. waiting for couchbase server to come up ... [2021-03-25 08:43:43,518] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.70: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-03-25 08:43:43,562] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:43:43,562] - [remote_util] [140232468973312] - INFO - Couchbase server status: [' Process: 99562 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-03-25 08:43:43,562] - [remote_util] [140232468973312] - INFO - Checking for process beam.smp on linux [2021-03-25 08:43:43,630] - [basetestcase] [140232468973312] - INFO - Couchbase started [2021-03-25 08:43:43,745] - [basetestcase] [140232468973312] - INFO - sleep for 10 secs. ... [2021-03-25 08:43:54,190] - [task] [140229784618752] - INFO - server: ip:172.23.123.148 port:8091 ssh_username:root, nodes/self [2021-03-25 08:43:54,196] - [task] [140229784618752] - INFO - {'ip': u'172.23.123.148', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.123.148', 'uptime': u'88', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.123.148:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3709796352, 'memoryTotal': 4201672704, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:43:54,201] - [rest_client] [140229784618752] - ERROR - GET http://172.23.123.148:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:43:54,201] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:43:54,209] - [rest_client] [140229784618752] - INFO - --> init_node_services(Administrator,password,172.23.123.148,8091,['kv']) [2021-03-25 08:43:54,209] - [rest_client] [140229784618752] - INFO - /node/controller/setupServices params on 172.23.123.148: 8091:services=kv&password=password&hostname=172.23.123.148&user=Administrator [2021-03-25 08:43:54,218] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.123.148:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:43:54,264] - [remote_util] [140229784618752] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:43:54,400] - [remote_util] [140229784618752] - INFO - Connected to 172.23.123.148 [2021-03-25 08:43:54,696] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:55,126] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:55,139] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.123.148: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:43:55,206] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:43:55,210] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.123.148:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:55,215] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.123.148:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:55,215] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:43:55,244] - [task] [140229784618752] - INFO - server: ip:172.23.123.150 port:8091 ssh_username:root, nodes/self [2021-03-25 08:43:55,249] - [task] [140229784618752] - INFO - {'ip': u'172.23.123.150', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.123.150', 'uptime': u'78', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.123.150:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3660541952, 'memoryTotal': 4201672704, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:43:55,254] - [rest_client] [140229784618752] - ERROR - GET http://172.23.123.150:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:43:55,254] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:43:55,262] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.123.150:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:43:55,307] - [remote_util] [140229784618752] - INFO - connecting to 172.23.123.150 with username:root [2021-03-25 08:43:55,441] - [remote_util] [140229784618752] - INFO - Connected to 172.23.123.150 [2021-03-25 08:43:55,740] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:56,172] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:56,185] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.123.150: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:43:56,252] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:43:56,256] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.123.150:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:56,261] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.123.150:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:56,262] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:43:56,300] - [task] [140229784618752] - INFO - server: ip:172.23.105.224 port:8091 ssh_username:root, nodes/self [2021-03-25 08:43:56,308] - [task] [140229784618752] - INFO - {'ip': u'172.23.105.224', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.224', 'uptime': u'62', 'mcdMemoryReserved': 3023, 'storageTotalRam': 3779, 'hostname': u'172.23.105.224:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3489513472, 'memoryTotal': 3963314176, 'memoryQuota': 1601, 'mcdMemoryAllocated': 3023, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:43:56,313] - [rest_client] [140229784618752] - ERROR - GET http://172.23.105.224:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:43:56,314] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2025 [2021-03-25 08:43:56,327] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.105.224:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:43:56,430] - [remote_util] [140229784618752] - INFO - connecting to 172.23.105.224 with username:root [2021-03-25 08:43:56,604] - [remote_util] [140229784618752] - INFO - Connected to 172.23.105.224 [2021-03-25 08:43:57,367] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:58,110] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:58,127] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.105.224: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:43:58,278] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:43:58,283] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.105.224:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:58,288] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.105.224:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:58,288] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:43:58,324] - [task] [140229784618752] - INFO - server: ip:172.23.121.217 port:8091 ssh_username:root, nodes/self [2021-03-25 08:43:58,329] - [task] [140229784618752] - INFO - {'ip': u'172.23.121.217', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.121.217', 'uptime': u'57', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.217:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3657170944, 'memoryTotal': 4201689088, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:43:58,334] - [rest_client] [140229784618752] - ERROR - GET http://172.23.121.217:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:43:58,334] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:43:58,341] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.121.217:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:43:58,383] - [remote_util] [140229784618752] - INFO - connecting to 172.23.121.217 with username:root [2021-03-25 08:43:58,516] - [remote_util] [140229784618752] - INFO - Connected to 172.23.121.217 [2021-03-25 08:43:58,785] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:43:59,210] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:43:59,222] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.121.217: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:43:59,289] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:43:59,293] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.217:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:59,298] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.217:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:43:59,298] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:43:59,324] - [task] [140229784618752] - INFO - server: ip:172.23.121.219 port:8091 ssh_username:root, nodes/self [2021-03-25 08:43:59,329] - [task] [140229784618752] - INFO - {'ip': u'172.23.121.219', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.121.219', 'uptime': u'48', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.219:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3681218560, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:43:59,333] - [rest_client] [140229784618752] - ERROR - GET http://172.23.121.219:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:43:59,334] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:43:59,341] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.121.219:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:43:59,384] - [remote_util] [140229784618752] - INFO - connecting to 172.23.121.219 with username:root [2021-03-25 08:43:59,518] - [remote_util] [140229784618752] - INFO - Connected to 172.23.121.219 [2021-03-25 08:43:59,833] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:44:00,260] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:44:00,273] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.121.219: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:44:00,340] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:44:00,345] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.219:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:00,349] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.219:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:00,349] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:44:00,382] - [task] [140229784618752] - INFO - server: ip:172.23.121.210 port:8091 ssh_username:root, nodes/self [2021-03-25 08:44:00,387] - [task] [140229784618752] - INFO - {'ip': u'172.23.121.210', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'37', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.210:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3635707904, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:44:00,391] - [rest_client] [140229784618752] - ERROR - GET http://172.23.121.210:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:44:00,392] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:44:00,400] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.121.210:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:44:00,443] - [remote_util] [140229784618752] - INFO - connecting to 172.23.121.210 with username:root [2021-03-25 08:44:00,579] - [remote_util] [140229784618752] - INFO - Connected to 172.23.121.210 [2021-03-25 08:44:00,871] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:44:01,299] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:44:01,312] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.121.210: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:44:01,378] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:44:01,384] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.210:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:01,389] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.210:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:01,389] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:44:01,416] - [task] [140229784618752] - INFO - server: ip:172.23.121.216 port:8091 ssh_username:root, nodes/self [2021-03-25 08:44:01,422] - [task] [140229784618752] - INFO - {'ip': u'172.23.121.216', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'27', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.216:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3643875328, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:44:01,426] - [rest_client] [140229784618752] - ERROR - GET http://172.23.121.216:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:44:01,426] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:44:01,434] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.121.216:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:44:01,482] - [remote_util] [140229784618752] - INFO - connecting to 172.23.121.216 with username:root [2021-03-25 08:44:01,616] - [remote_util] [140229784618752] - INFO - Connected to 172.23.121.216 [2021-03-25 08:44:01,907] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:44:02,334] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:44:02,347] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.121.216: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:44:02,414] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:44:02,418] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.216:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:02,422] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.216:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:02,423] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:44:02,450] - [task] [140229784618752] - INFO - server: ip:172.23.121.70 port:8091 ssh_username:root, nodes/self [2021-03-25 08:44:02,455] - [task] [140229784618752] - INFO - {'ip': u'172.23.121.70', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'18', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.70:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.6.2-9577-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393222, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3651198976, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2021-03-25 08:44:02,460] - [rest_client] [140229784618752] - ERROR - GET http://172.23.121.70:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2021-03-25 08:44:02,461] - [rest_client] [140229784618752] - INFO - pools/default params : memoryQuota=2147 [2021-03-25 08:44:02,468] - [rest_client] [140229784618752] - INFO - settings/web params on 172.23.121.70:8091:username=Administrator&password=password&port=8091 [2021-03-25 08:44:02,511] - [remote_util] [140229784618752] - INFO - connecting to 172.23.121.70 with username:root [2021-03-25 08:44:02,645] - [remote_util] [140229784618752] - INFO - Connected to 172.23.121.70 [2021-03-25 08:44:02,950] - [remote_util] [140229784618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:44:03,378] - [remote_util] [140229784618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:44:03,394] - [remote_util] [140229784618752] - INFO - running command.raw on 172.23.121.70: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:44:03,460] - [remote_util] [140229784618752] - INFO - command executed successfully [2021-03-25 08:44:03,464] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.70:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:03,469] - [rest_client] [140229784618752] - INFO - /diag/eval status on 172.23.121.70:8091: True content: [6,6] command: cluster_compat_mode:get_compat_version(). [2021-03-25 08:44:03,469] - [rest_client] [140229784618752] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:44:03,503] - [basetestcase] [140232468973312] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.148 **** [2021-03-25 08:44:03,523] - [rest_client] [140232468973312] - ERROR - DELETE http://172.23.123.148:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2021-03-25 08:44:03,524] - [internal_user] [140232468973312] - INFO - Exception while deleting user. Exception is -"User was not found." [2021-03-25 08:44:03,577] - [basetestcase] [140232468973312] - INFO - sleep for 5 secs. ... [2021-03-25 08:44:08,583] - [basetestcase] [140232468973312] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-03-25 08:44:08,603] - [basetestcase] [140232468973312] - INFO - done initializing cluster [2021-03-25 08:44:09,496] - [task] [140229784618752] - INFO - adding node 172.23.123.150:8091 to cluster [2021-03-25 08:44:09,497] - [rest_client] [140229784618752] - INFO - adding remote node @172.23.123.150:8091 to this cluster @172.23.123.148:8091 [2021-03-25 08:44:11,262] - [task] [140229784618752] - INFO - adding node 172.23.105.224:8091 to cluster [2021-03-25 08:44:11,262] - [rest_client] [140229784618752] - INFO - adding remote node @172.23.105.224:8091 to this cluster @172.23.123.148:8091 [2021-03-25 08:44:15,298] - [task] [140229784618752] - INFO - adding node 172.23.121.217:8091 to cluster [2021-03-25 08:44:15,299] - [rest_client] [140229784618752] - INFO - adding remote node @172.23.121.217:8091 to this cluster @172.23.123.148:8091 [2021-03-25 08:44:16,953] - [task] [140229784618752] - INFO - adding node 172.23.121.219:8091 to cluster [2021-03-25 08:44:16,953] - [rest_client] [140229784618752] - INFO - adding remote node @172.23.121.219:8091 to this cluster @172.23.123.148:8091 [2021-03-25 08:44:18,573] - [rest_client] [140229784618752] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.121.217,ns_1@172.23.121.219,ns_1@172.23.105.224,ns_1@172.23.123.150,ns_1@172.23.123.148'} [2021-03-25 08:44:18,582] - [rest_client] [140229784618752] - INFO - rebalance operation started [2021-03-25 08:44:18,587] - [rest_client] [140229784618752] - INFO - rebalance percentage : 0.00 % [2021-03-25 08:44:18,588] - [task] [140229784618752] - INFO - Rebalance - status: running, progress: 0.00% [2021-03-25 08:44:28,616] - [task] [140229784618752] - INFO - Rebalance - status: none, progress: 100.00% [2021-03-25 08:44:28,626] - [task] [140229784618752] - INFO - rebalancing was completed with progress: 100% in 10.0432372093 sec [2021-03-25 08:44:28,629] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:44:28,765] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.148 [2021-03-25 08:44:29,074] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:44:29,482] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:44:29,495] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.123.148: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-03-25 08:44:29,562] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:44:29,562] - [basetestcase] [140232468973312] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.148 [2021-03-25 08:44:29,563] - [basetestcase] [140232468973312] - INFO - ============== basetestcase setup was finished for test #2 test_vb_shuffle_during_failover_add_back ============== [2021-03-25 08:44:29,588] - [basetestcase] [140232468973312] - INFO - ------- Cluster statistics ------- [2021-03-25 08:44:29,588] - [basetestcase] [140232468973312] - INFO - 172.23.121.217:8091 => {'swap_mem_used': 128188416, 'cpu_utilization': 0.5, 'mem_free': 3601051648, 'swap_mem_total': 3758092288, 'mem_total': 4201689088, 'services': [u'eventing']} [2021-03-25 08:44:29,588] - [basetestcase] [140232468973312] - INFO - 172.23.121.219:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.5012531328320802, 'mem_free': 3594518528, 'swap_mem_total': 3758092288, 'mem_total': 4201684992, 'services': [u'index', u'n1ql']} [2021-03-25 08:44:29,588] - [basetestcase] [140232468973312] - INFO - 172.23.105.224:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.7556675062972292, 'mem_free': 3433742336, 'swap_mem_total': 3758092288, 'mem_total': 3963314176, 'services': [u'eventing']} [2021-03-25 08:44:29,589] - [basetestcase] [140232468973312] - INFO - 172.23.123.150:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3617841152, 'swap_mem_total': 3758092288, 'mem_total': 4201672704, 'services': [u'kv']} [2021-03-25 08:44:29,589] - [basetestcase] [140232468973312] - INFO - 172.23.123.148:8091 => {'swap_mem_used': 113246208, 'cpu_utilization': 0.5, 'mem_free': 3665731584, 'swap_mem_total': 3758092288, 'mem_total': 4201672704, 'services': [u'kv']} [2021-03-25 08:44:29,589] - [basetestcase] [140232468973312] - INFO - --- End of cluster statistics --- [2021-03-25 08:44:29,591] - [remote_util] [140232468973312] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:44:29,725] - [remote_util] [140232468973312] - INFO - Connected to 172.23.123.148 [2021-03-25 08:44:30,025] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:44:30,463] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:44:37,319] - [basetestcase] [140232468973312] - INFO - list of n1ql nodes in cluster: [ip:172.23.121.219 port:8091 ssh_username:root] [2021-03-25 08:44:37,332] - [basetestcase] [140232468973312] - INFO - list of eventing nodes in cluster: [ip:172.23.121.217 port:8091 ssh_username:root, ip:172.23.105.224 port:8091 ssh_username:root] [2021-03-25 08:44:37,339] - [rest_client] [140232468973312] - INFO - settings/indexes params : storageMode=plasma [2021-03-25 08:44:37,347] - [eventing_base] [140232468973312] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-03-25 08:44:37,347] - [rest_client] [140232468973312] - INFO - pools/default params : memoryQuota=330 [2021-03-25 08:44:37,356] - [rest_client] [140232468973312] - INFO - pools/default params : indexMemoryQuota=256 [2021-03-25 08:44:37,364] - [rest_client] [140232468973312] - INFO - pools/default params : eventingMemoryQuota=512 [2021-03-25 08:44:37,387] - [basetestcase] [140232468973312] - INFO - list of n1ql nodes in cluster: [ip:172.23.121.219 port:8091 ssh_username:root] [2021-03-25 08:44:37,388] - [rest_client] [140232468973312] - INFO - pools/default params : memoryQuota=700 [2021-03-25 08:44:37,395] - [eventing_failover] [140232468973312] - INFO - 100 [2021-03-25 08:44:37,680] - [rest_client] [140229784618752] - INFO - http://172.23.123.148:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=100&authType=none&name=src_bucket&storageBackend=couchstore&flushEnabled=1&replicaNumber=0&compressionMode=passive&replicaIndex=1 [2021-03-25 08:44:37,688] - [rest_client] [140229784618752] - INFO - 0.01 seconds to create bucket src_bucket [2021-03-25 08:44:37,689] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:38,340] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:38,581] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:38,701] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:38,783] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:39,003] - [task] [140229784618752] - WARNING - vbucket map not ready after try 0 [2021-03-25 08:44:39,003] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:39,116] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:39,199] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:39,464] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:39,544] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:39,760] - [task] [140229784618752] - WARNING - vbucket map not ready after try 1 [2021-03-25 08:44:39,760] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:39,871] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:39,954] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:40,234] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:40,319] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:40,384] - [task] [140229784618752] - WARNING - vbucket map not ready after try 2 [2021-03-25 08:44:40,384] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:40,647] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:40,728] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:40,984] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:41,065] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:41,129] - [task] [140229784618752] - WARNING - vbucket map not ready after try 3 [2021-03-25 08:44:41,130] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:41,404] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:41,486] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:41,756] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:41,844] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:41,905] - [task] [140229784618752] - WARNING - vbucket map not ready after try 4 [2021-03-25 08:44:41,905] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:42,165] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:42,246] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:42,511] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 src_bucket [2021-03-25 08:44:42,592] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 src_bucket [2021-03-25 08:44:42,660] - [task] [140229784618752] - WARNING - vbucket map not ready after try 5 [2021-03-25 08:44:43,703] - [rest_client] [140229784618752] - INFO - http://172.23.123.148:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=100&authType=none&name=dst_bucket&storageBackend=couchstore&flushEnabled=1&replicaNumber=0&compressionMode=passive&replicaIndex=1 [2021-03-25 08:44:43,713] - [rest_client] [140229784618752] - INFO - 0.01 seconds to create bucket dst_bucket [2021-03-25 08:44:43,713] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:44,532] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:44,620] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:44,905] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:44,991] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:45,059] - [task] [140229784618752] - WARNING - vbucket map not ready after try 0 [2021-03-25 08:44:45,059] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:45,327] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:45,413] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:45,701] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:45,789] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:45,859] - [task] [140229784618752] - WARNING - vbucket map not ready after try 1 [2021-03-25 08:44:45,860] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:46,126] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:46,217] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:46,343] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:46,579] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:46,647] - [task] [140229784618752] - WARNING - vbucket map not ready after try 2 [2021-03-25 08:44:46,648] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:46,772] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:47,012] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:47,137] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:47,376] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:47,448] - [task] [140229784618752] - WARNING - vbucket map not ready after try 3 [2021-03-25 08:44:47,449] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:47,568] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:47,809] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:47,932] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:48,174] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:48,250] - [task] [140229784618752] - WARNING - vbucket map not ready after try 4 [2021-03-25 08:44:48,251] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.148 to accept set ops [2021-03-25 08:44:48,372] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:48,616] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:48,742] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 dst_bucket [2021-03-25 08:44:48,842] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 dst_bucket [2021-03-25 08:44:49,069] - [task] [140229784618752] - WARNING - vbucket map not ready after try 5 [2021-03-25 08:44:50,111] - [rest_client] [140229784618752] - INFO - http://172.23.123.148:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=200&authType=none&name=metadata&storageBackend=couchstore&flushEnabled=1&replicaNumber=0&compressionMode=passive&replicaIndex=1 [2021-03-25 08:44:50,121] - [rest_client] [140229784618752] - INFO - 0.01 seconds to create bucket metadata [2021-03-25 08:44:50,121] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.148 to accept set ops [2021-03-25 08:44:50,786] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:51,031] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:51,161] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:51,249] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:51,476] - [task] [140229784618752] - WARNING - vbucket map not ready after try 0 [2021-03-25 08:44:51,476] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.148 to accept set ops [2021-03-25 08:44:51,601] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:51,688] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:51,971] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:52,056] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:52,288] - [task] [140229784618752] - WARNING - vbucket map not ready after try 1 [2021-03-25 08:44:52,289] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.148 to accept set ops [2021-03-25 08:44:52,405] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:52,495] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:52,770] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:52,861] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:52,925] - [task] [140229784618752] - WARNING - vbucket map not ready after try 2 [2021-03-25 08:44:52,926] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.148 to accept set ops [2021-03-25 08:44:53,203] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:53,289] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:53,570] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:53,657] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:53,726] - [task] [140229784618752] - WARNING - vbucket map not ready after try 3 [2021-03-25 08:44:53,727] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.148 to accept set ops [2021-03-25 08:44:53,999] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:54,083] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:54,209] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:54,452] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:54,526] - [task] [140229784618752] - WARNING - vbucket map not ready after try 4 [2021-03-25 08:44:54,526] - [bucket_helper] [140229784618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.148 to accept set ops [2021-03-25 08:44:54,642] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:54,874] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:54,992] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.148:11210 metadata [2021-03-25 08:44:55,233] - [data_helper] [140229784618752] - INFO - creating direct client 172.23.123.150:11210 metadata [2021-03-25 08:44:55,301] - [task] [140229784618752] - WARNING - vbucket map not ready after try 5 [2021-03-25 08:44:55,619] - [basetestcase] [140232468973312] - INFO - list of n1ql nodes in cluster: [ip:172.23.121.219 port:8091 ssh_username:root] [2021-03-25 08:44:55,619] - [tuq_helper] [140232468973312] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-03-25 08:44:55,627] - [rest_client] [140232468973312] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-03-25 08:44:55,826] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 196.537349ms [2021-03-25 08:44:55,826] - [tuq_helper] [140232468973312] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2021-03-25 08:44:55,837] - [rest_client] [140232468973312] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2021-03-25 08:44:57,622] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 1.782312304s [2021-03-25 08:44:57,623] - [tuq_helper] [140232468973312] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-03-25 08:44:57,634] - [rest_client] [140232468973312] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-03-25 08:44:57,664] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 25.464771ms [2021-03-25 08:44:57,664] - [tuq_helper] [140232468973312] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-03-25 08:44:57,671] - [rest_client] [140232468973312] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-03-25 08:44:57,697] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 22.862835ms [2021-03-25 08:44:57,697] - [tuq_helper] [140232468973312] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2021-03-25 08:44:57,706] - [rest_client] [140232468973312] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2021-03-25 08:44:59,399] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 1.690270345s [2021-03-25 08:44:59,400] - [tuq_helper] [140232468973312] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-03-25 08:44:59,411] - [rest_client] [140232468973312] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-03-25 08:44:59,448] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 33.374725ms [2021-03-25 08:44:59,448] - [tuq_helper] [140232468973312] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-03-25 08:44:59,455] - [rest_client] [140232468973312] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-03-25 08:44:59,480] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 22.858236ms [2021-03-25 08:44:59,480] - [tuq_helper] [140232468973312] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2021-03-25 08:44:59,487] - [rest_client] [140232468973312] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2021-03-25 08:45:01,167] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 1.676413185s [2021-03-25 08:45:01,170] - [tuq_helper] [140232468973312] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-03-25 08:45:01,179] - [rest_client] [140232468973312] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-03-25 08:45:01,207] - [tuq_helper] [140232468973312] - INFO - TOTAL ELAPSED TIME: 25.430237ms [2021-03-25 08:45:01,229] - [basetestcase] [140232468973312] - INFO - list of eventing nodes in cluster: [ip:172.23.121.217 port:8091 ssh_username:root, ip:172.23.105.224 port:8091 ssh_username:root] [2021-03-25 08:45:01,229] - [eventing_base] [140232468973312] - INFO - Deploying the following handler code : Function_60783645_test_vb_shuffle_during_failover_add_back with bindings: {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [], 'metadata_bucket': 'metadata'} and settings: {'cleanup_timers': False, 'worker_count': 3, 'skip_timer_threshold': 86400, 'dcp_stream_boundary': 'everything', 'use_memory_manager': True, 'sock_batch_size': 1, 'language_compatibility': '6.6.2', 'log_level': 'INFO', 'tick_duration': 5000, 'deadline_timeout': 21, 'timer_worker_pool_size': 3, 'deployment_status': True, 'description': 'Sample Description', 'dcp_gen_chan_size': 10000, 'timer_storage_chan_size': 10000, 'worker_queue_cap': 100000, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'processing_status': True, 'timer_processing_tick_interval': 500, 'data_chan_size': 10000, 'checkpoint_interval': 20000} [2021-03-25 08:45:01,229] - [eventing_base] [140232468973312] - INFO - function OnUpdate(doc, meta) { log('document', doc); try { var time_rand = random_gen(); dst_bucket[meta.id + time_rand] = doc; } catch(e) { log(e); //var time_rand = random_gen(); //dst_bucket[meta.id + time_rand] = doc; } } function random_gen(){ var rand = Math.floor(Math.random() * 20000000) * Math.floor(Math.random() * 20000000); var time_rand = Math.round((new Date()).getTime() / 1000) + rand; return time_rand; } [2021-03-25 08:45:02,315] - [eventing_base] [140232468973312] - INFO - deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_60783645_test_vb_shuffle_during_failover_add_back' in metakv", "warnings": null } } [2021-03-25 08:45:02,316] - [basetestcase] [140232468973312] - INFO - sleep for 20 secs. Waiting for Function_60783645_test_vb_shuffle_during_failover_add_back to deployed... ... [2021-03-25 08:45:22,356] - [basetestcase] [140232468973312] - INFO - sleep for 20 secs. Waiting for Function_60783645_test_vb_shuffle_during_failover_add_back to deployed... ... [2021-03-25 08:45:42,382] - [rest_client] [140232468973312] - ERROR - GET http://172.23.121.217:8096//api/v1/status body: headers: {'Content-type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 500 reason: unknown { "name": "ERR_FETCHING_ACTIVE_EVENTING_NODES", "code": 37, "description": "Failed to fetch active Eventing nodes", "attributes": null, "runtime_info": { "code": 37, "info": "Unable to fetch active Eventing nodes, err: Get http://172.23.105.224:8096/uuid: dial tcp 172.23.105.224:8096: connect: connection refused" } } auth: Administrator:password [2021-03-25 08:45:42,405] - [basetestcase] [140232468973312] - INFO - list of eventing nodes in cluster: [ip:172.23.121.217 port:8091 ssh_username:root, ip:172.23.105.224 port:8091 ssh_username:root] [2021-03-25 08:45:42,429] - [eventing_base] [140232468973312] - INFO - Stats for Node 172.23.121.217 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 968, "agg_queue_memory": 0, "agg_queue_memory_cap": 267386880, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 87, "failure_stats": 87, "latency_stats": 87, "lcb_exception_stats": 87, "log_level": 3, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 956, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104907": "2021-03-25T15:45:41Z", "104917": "2021-03-25T15:45:41Z", "104923": "2021-03-25T15:45:41Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104907": "2021-03-25T15:45:41Z", "104917": "2021-03-25T15:45:41Z", "104923": "2021-03-25T15:45:41Z" }, "v8worker_events_lost": 0 }, "function_id": 712793564, "function_name": "Function_60783645_test_vb_shuffle_during_failover_add_back", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": "[512-682]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": "[683-853]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": "[854-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 6, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.224:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.121.217:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.105.224:8096": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": "[0-170]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": "[171-341]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": "[342-511]" }, "172.23.121.217:8096": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": "[512-682]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": "[683-853]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": "[854-1023]" } }, "worker_pids": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": 104907, "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": 104917, "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": 104923 } } ] [2021-03-25 08:45:42,509] - [eventing_base] [140232468973312] - INFO - Stats for Node 172.23.105.224 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 968, "agg_queue_memory": 0, "agg_queue_memory_cap": 267386880, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_stream_req_counter": 512, "dcp_streamreq": 512, "execution_stats": 87, "failure_stats": 87, "latency_stats": 87, "lcb_exception_stats": 87, "log_level": 3, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 956, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "22412": "2021-03-25T15:45:41Z", "22418": "2021-03-25T15:45:41Z", "22433": "2021-03-25T15:45:41Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "22412": "2021-03-25T15:45:41Z", "22418": "2021-03-25T15:45:41Z", "22433": "2021-03-25T15:45:41Z" }, "v8worker_events_lost": 0 }, "function_id": 712793564, "function_name": "Function_60783645_test_vb_shuffle_during_failover_add_back", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": "[0-170]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": "[171-341]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": "[342-511]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 6, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.224:8096", "start_vb": 0, "vb_count": 512 }, { "host_name": "172.23.121.217:8096", "start_vb": 512, "vb_count": 512 } ], "vb_distribution_stats_from_metadata": { "172.23.105.224:8096": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": "[0-170]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": "[171-341]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": "[342-511]" }, "172.23.121.217:8096": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": "[512-682]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": "[683-853]", "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": "[854-1023]" } }, "worker_pids": { "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_0": 22412, "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_1": 22418, "worker_Function_60783645_test_vb_shuffle_during_failover_add_back_2": 22433 } } ] [2021-03-25 08:45:42,535] - [basetestcase] [140232468973312] - INFO - list of eventing nodes in cluster: [ip:172.23.121.217 port:8091 ssh_username:root, ip:172.23.105.224 port:8091 ssh_username:root] [2021-03-25 08:45:42,537] - [remote_util] [140232468973312] - INFO - connecting to 172.23.121.217 with username:root [2021-03-25 08:45:42,670] - [remote_util] [140232468973312] - INFO - Connected to 172.23.121.217 [2021-03-25 08:45:42,935] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:45:43,364] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:45:43,378] - [rest_client] [140232468973312] - INFO - /diag/eval status on 172.23.121.217:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-03-25 08:45:43,378] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-03-25 08:45:43,476] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:45:43,477] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.121.217: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-03-25 08:45:43,511] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:45:43,627] - [remote_util] [140232468973312] - INFO - connecting to 172.23.105.224 with username:root [2021-03-25 08:45:57,892] - [remote_util] [140232468973312] - INFO - Connected to 172.23.105.224 [2021-03-25 08:45:58,629] - [remote_util] [140232468973312] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:45:59,455] - [remote_util] [140232468973312] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:45:59,482] - [rest_client] [140232468973312] - INFO - /diag/eval status on 172.23.105.224:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-03-25 08:45:59,482] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-03-25 08:45:59,609] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:45:59,609] - [remote_util] [140232468973312] - INFO - running command.raw on 172.23.105.224: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-03-25 08:45:59,794] - [remote_util] [140232468973312] - INFO - command executed successfully [2021-03-25 08:45:59,948] - [eventing_base] [140232468973312] - INFO - Bucket dst_bucket DGM is 100 [2021-03-25 08:45:59,989] - [eventing_base] [140232468973312] - INFO - Bucket metadata DGM is 100 [2021-03-25 08:46:00,025] - [eventing_base] [140232468973312] - INFO - Bucket src_bucket DGM is 100 [2021-03-25 08:46:00,065] - [eventing_base] [140232468973312] - INFO - number of documents in metadata bucket 1024 [2021-03-25 08:46:00,073] - [remote_util] [140235187218240] - INFO - connecting to 172.23.123.148 with username:root [2021-03-25 08:46:00,207] - [remote_util] [140235187218240] - INFO - Connected to 172.23.123.148 [2021-03-25 08:46:00,483] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:46:00,915] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:46:00,916] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.123.148: /opt/couchbase/bin/cbcollect_info 172.23.123.148-20210325-0846-diag.zip [2021-03-25 08:48:33,522] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:48:34,195] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.123.148: rm -f /root/172.23.123.148-20210325-0846-diag.zip [2021-03-25 08:48:34,296] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:48:34,413] - [remote_util] [140235187218240] - INFO - connecting to 172.23.123.150 with username:root [2021-03-25 08:48:34,547] - [remote_util] [140235187218240] - INFO - Connected to 172.23.123.150 [2021-03-25 08:48:34,842] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:48:35,257] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:48:35,257] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.123.150: /opt/couchbase/bin/cbcollect_info 172.23.123.150-20210325-0848-diag.zip [2021-03-25 08:51:03,834] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:51:04,578] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.123.150: rm -f /root/172.23.123.150-20210325-0848-diag.zip [2021-03-25 08:51:04,617] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:51:04,735] - [remote_util] [140235187218240] - INFO - connecting to 172.23.105.224 with username:root [2021-03-25 08:51:04,902] - [remote_util] [140235187218240] - INFO - Connected to 172.23.105.224 [2021-03-25 08:51:05,670] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:51:06,569] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:51:06,570] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.105.224: /opt/couchbase/bin/cbcollect_info 172.23.105.224-20210325-0851-diag.zip [2021-03-25 08:52:51,867] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:52:52,448] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.105.224: rm -f /root/172.23.105.224-20210325-0851-diag.zip [2021-03-25 08:52:52,575] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:52:52,642] - [remote_util] [140235187218240] - INFO - connecting to 172.23.121.217 with username:root [2021-03-25 08:52:52,776] - [remote_util] [140235187218240] - INFO - Connected to 172.23.121.217 [2021-03-25 08:52:53,046] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:52:53,475] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:52:53,476] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.217: /opt/couchbase/bin/cbcollect_info 172.23.121.217-20210325-0852-diag.zip [2021-03-25 08:54:18,939] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:54:19,519] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.217: rm -f /root/172.23.121.217-20210325-0852-diag.zip [2021-03-25 08:54:19,602] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:54:19,719] - [remote_util] [140235187218240] - INFO - connecting to 172.23.121.219 with username:root [2021-03-25 08:54:19,853] - [remote_util] [140235187218240] - INFO - Connected to 172.23.121.219 [2021-03-25 08:54:20,143] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:54:20,572] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:54:20,573] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.219: /opt/couchbase/bin/cbcollect_info 172.23.121.219-20210325-0854-diag.zip [2021-03-25 08:56:12,638] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:56:13,077] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.219: rm -f /root/172.23.121.219-20210325-0854-diag.zip [2021-03-25 08:56:13,160] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:56:13,277] - [remote_util] [140235187218240] - INFO - connecting to 172.23.121.210 with username:root [2021-03-25 08:56:13,411] - [remote_util] [140235187218240] - INFO - Connected to 172.23.121.210 [2021-03-25 08:56:13,702] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:56:14,134] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:56:14,135] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.210: /opt/couchbase/bin/cbcollect_info 172.23.121.210-20210325-0856-diag.zip [2021-03-25 08:57:35,407] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:57:35,870] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.210: rm -f /root/172.23.121.210-20210325-0856-diag.zip [2021-03-25 08:57:35,953] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:57:36,070] - [remote_util] [140235187218240] - INFO - connecting to 172.23.121.216 with username:root [2021-03-25 08:57:36,204] - [remote_util] [140235187218240] - INFO - Connected to 172.23.121.216 [2021-03-25 08:57:36,499] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:57:36,928] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:57:36,928] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.216: /opt/couchbase/bin/cbcollect_info 172.23.121.216-20210325-0857-diag.zip [2021-03-25 08:59:04,159] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:59:04,787] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.216: rm -f /root/172.23.121.216-20210325-0857-diag.zip [2021-03-25 08:59:04,870] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 08:59:04,986] - [remote_util] [140235187218240] - INFO - connecting to 172.23.121.70 with username:root [2021-03-25 08:59:05,120] - [remote_util] [140235187218240] - INFO - Connected to 172.23.121.70 [2021-03-25 08:59:05,413] - [remote_util] [140235187218240] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-03-25 08:59:05,842] - [remote_util] [140235187218240] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: CentOS 7 [2021-03-25 08:59:05,843] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.70: /opt/couchbase/bin/cbcollect_info 172.23.121.70-20210325-0859-diag.zip [2021-03-25 09:00:27,193] - [remote_util] [140235187218240] - INFO - command executed successfully [2021-03-25 09:00:27,650] - [remote_util] [140235187218240] - INFO - running command.raw on 172.23.121.70: rm -f /root/172.23.121.70-20210325-0859-diag.zip [2021-03-25 09:00:27,732] - [remote_util] [140235187218240] - INFO - command executed successfully