[2020-01-22 15:26:28,555] - [eventing_base] [140650546005760] - INFO - Starting Test: test_killing_eventing_processes_during_eventing_rebalance [2020-01-22 15:26:28,560] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:26:28,873] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:26:29,112] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:29,593] - [rest_client] [140650546005760] - INFO - Node version in cluster 6.0.4-3070-enterprise [2020-01-22 15:26:29,603] - [rest_client] [140650546005760] - ERROR - GET http://172.23.108.212:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-01-22 15:26:29,603] - [rest_client] [140650546005760] - INFO - Node versions in cluster [] [2020-01-22 15:26:29,604] - [basetestcase] [140650546005760] - INFO - ============== basetestcase setup was started for test #28 test_killing_eventing_processes_during_eventing_rebalance============== [2020-01-22 15:26:29,613] - [rest_client] [140650546005760] - ERROR - GET http://172.23.108.212:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2020-01-22 15:26:29,762] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.108.212:8091 [2020-01-22 15:26:29,767] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.108.212:8091 is running [2020-01-22 15:26:29,773] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.121.238:8091 [2020-01-22 15:26:29,778] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.121.238:8091 is running [2020-01-22 15:26:29,783] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.121.241:8091 [2020-01-22 15:26:29,788] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.121.241:8091 is running [2020-01-22 15:26:29,793] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.121.91:8091 [2020-01-22 15:26:29,797] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.121.91:8091 is running [2020-01-22 15:26:29,802] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.121.95:8091 [2020-01-22 15:26:29,806] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.121.95:8091 is running [2020-01-22 15:26:29,811] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.122.62:8091 [2020-01-22 15:26:29,816] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.122.62:8091 is running [2020-01-22 15:26:29,821] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.122.73:8091 [2020-01-22 15:26:29,825] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.122.73:8091 is running [2020-01-22 15:26:29,829] - [cluster_helper] [140650546005760] - INFO - waiting for ns_server @ 172.23.97.220:8091 [2020-01-22 15:26:29,834] - [cluster_helper] [140650546005760] - INFO - ns_server @ 172.23.97.220:8091 is running [2020-01-22 15:26:29,835] - [basetestcase] [140650546005760] - INFO - initializing cluster [2020-01-22 15:26:29,836] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:26:30,121] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:26:30,323] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:30,763] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:26:31,057] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:26:31,267] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:31,735] - [remote_util] [140650546005760] - INFO - 172.23.108.212 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:31,819] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:31,820] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: systemctl stop couchbase-server.service [2020-01-22 15:26:35,294] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:35,294] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:26:35,408] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:26:35,445] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:35,445] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:26:35,511] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:35,513] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:26:35,814] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:26:36,049] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:36,558] - [remote_util] [140650546005760] - INFO - 172.23.108.212 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:36,649] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:36,649] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: systemctl start couchbase-server.service [2020-01-22 15:26:36,716] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:36,716] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:26:36,832] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:26:37,088] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:26:37,411] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:37,845] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:26:38,101] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:26:38,397] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:38,903] - [remote_util] [140650546005760] - INFO - 172.23.121.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:39,029] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:39,029] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: systemctl stop couchbase-server.service [2020-01-22 15:26:41,852] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:41,853] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:26:41,967] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:26:41,986] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:41,986] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:26:42,067] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:42,069] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:26:42,329] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:26:42,629] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:43,136] - [remote_util] [140650546005760] - INFO - 172.23.121.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:43,194] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:43,194] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: systemctl start couchbase-server.service [2020-01-22 15:26:43,229] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:43,229] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:26:43,345] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:26:43,606] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:26:43,901] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:44,363] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:26:44,625] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:26:44,920] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:45,421] - [remote_util] [140650546005760] - INFO - 172.23.121.241 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:45,540] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:45,540] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: systemctl stop couchbase-server.service [2020-01-22 15:26:48,412] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:48,413] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:26:48,527] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:26:48,545] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:48,545] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:26:48,627] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:48,628] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:26:48,887] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:26:49,207] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:49,711] - [remote_util] [140650546005760] - INFO - 172.23.121.241 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:49,832] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:49,833] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: systemctl start couchbase-server.service [2020-01-22 15:26:49,867] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:49,867] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:26:49,983] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:26:50,242] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:26:50,563] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:50,999] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:26:51,256] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:26:51,514] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:52,021] - [remote_util] [140650546005760] - INFO - 172.23.121.91 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:52,087] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:52,087] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: systemctl stop couchbase-server.service [2020-01-22 15:26:54,859] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:54,860] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:26:54,974] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:26:54,993] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:54,993] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:26:55,075] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:55,077] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:26:55,334] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:26:55,653] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:56,158] - [remote_util] [140650546005760] - INFO - 172.23.121.91 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:56,281] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:56,281] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: systemctl start couchbase-server.service [2020-01-22 15:26:56,315] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:26:56,316] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:26:56,432] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:26:56,724] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:26:57,009] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:57,450] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:26:57,708] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:26:58,004] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:26:58,507] - [remote_util] [140650546005760] - INFO - 172.23.121.95 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:26:58,565] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:26:58,565] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: systemctl stop couchbase-server.service [2020-01-22 15:27:01,538] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:01,538] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:27:01,652] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:27:01,672] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:01,672] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:27:01,753] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:01,755] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:27:02,015] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:27:02,314] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:02,822] - [remote_util] [140650546005760] - INFO - 172.23.121.95 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:02,881] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:02,881] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: systemctl start couchbase-server.service [2020-01-22 15:27:02,916] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:02,916] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:27:03,032] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:27:03,292] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:27:03,608] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:04,053] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:27:04,310] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:27:04,629] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:05,131] - [remote_util] [140650546005760] - INFO - 172.23.122.62 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:05,186] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:05,186] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: systemctl stop couchbase-server.service [2020-01-22 15:27:08,159] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:08,160] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:27:08,224] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:27:08,242] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:08,243] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:27:08,324] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:08,325] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:27:08,579] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:27:08,897] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:09,395] - [remote_util] [140650546005760] - INFO - 172.23.122.62 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:09,512] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:09,512] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: systemctl start couchbase-server.service [2020-01-22 15:27:09,547] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:09,547] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:27:09,663] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:27:09,923] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:27:10,203] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:10,696] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:27:10,951] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:27:11,230] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:11,729] - [remote_util] [140650546005760] - INFO - 172.23.122.73 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:11,854] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:11,855] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: systemctl stop couchbase-server.service [2020-01-22 15:27:14,728] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:14,728] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:27:14,842] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:27:14,861] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:14,861] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:27:14,942] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:14,944] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:27:15,201] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:27:15,482] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:15,984] - [remote_util] [140650546005760] - INFO - 172.23.122.73 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:16,105] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:16,106] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: systemctl start couchbase-server.service [2020-01-22 15:27:16,140] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:16,140] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:27:16,256] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:27:16,507] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:27:16,688] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:17,198] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:27:17,438] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:27:17,607] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:18,070] - [remote_util] [140650546005760] - INFO - 172.23.97.220 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:18,174] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:18,174] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: systemctl stop couchbase-server.service [2020-01-22 15:27:21,047] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:21,047] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:27:21,161] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:27:21,180] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:21,180] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:27:21,260] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:21,263] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:27:21,511] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:27:21,680] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:22,213] - [remote_util] [140650546005760] - INFO - 172.23.97.220 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:27:22,318] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:27:22,318] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: systemctl start couchbase-server.service [2020-01-22 15:27:22,352] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:27:22,352] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:27:22,466] - [basetestcase] [140650546005760] - INFO - sleep for 10 secs. ... [2020-01-22 15:27:32,723] - [task] [140650554398464] - INFO - server: ip:172.23.108.212 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.108.212', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.108.212', 'uptime': u'47', 'mcdMemoryReserved': 3048, 'storageTotalRam': 3811, 'hostname': u'172.23.108.212:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3270623232, 'memoryTotal': 3996319744, 'memoryQuota': 1621, 'mcdMemoryAllocated': 3048, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:32,724] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2032 [2020-01-22 15:27:32,736] - [rest_client] [140650554398464] - INFO - /node/controller/setupServices params on 172.23.108.212: 8091:services=kv&password=password&hostname=172.23.108.212&user=Administrator [2020-01-22 15:27:32,752] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.108.212:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:32,825] - [remote_util] [140650554398464] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:27:33,149] - [remote_util] [140650554398464] - INFO - Connected to 172.23.108.212 [2020-01-22 15:27:33,405] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:33,897] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.108.212: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:33,971] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:33,977] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.108.212:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:33,982] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.108.212:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:33,982] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:34,094] - [task] [140650554398464] - INFO - server: ip:172.23.121.238 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.121.238', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.121.238', 'uptime': u'43', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.238:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3507773440, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:34,094] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2136 [2020-01-22 15:27:34,101] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.121.238:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:34,133] - [remote_util] [140650554398464] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:27:34,395] - [remote_util] [140650554398464] - INFO - Connected to 172.23.121.238 [2020-01-22 15:27:34,694] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:35,129] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.121.238: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:35,194] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:35,198] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.238:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:35,202] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.238:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:35,202] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:35,351] - [task] [140650554398464] - INFO - server: ip:172.23.121.241 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.121.241', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.121.241', 'uptime': u'38', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.241:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3586768896, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:35,351] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2136 [2020-01-22 15:27:35,357] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.121.241:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:35,390] - [remote_util] [140650554398464] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:27:35,650] - [remote_util] [140650554398464] - INFO - Connected to 172.23.121.241 [2020-01-22 15:27:35,970] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:36,426] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.121.241: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:36,491] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:36,496] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.241:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:36,499] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.241:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:36,500] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:36,648] - [task] [140650554398464] - INFO - server: ip:172.23.121.91 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.121.91', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.121.91', 'uptime': u'34', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.91:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3581263872, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:36,649] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2136 [2020-01-22 15:27:36,655] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.121.91:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:36,684] - [remote_util] [140650554398464] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:27:36,946] - [remote_util] [140650554398464] - INFO - Connected to 172.23.121.91 [2020-01-22 15:27:37,268] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:37,703] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.121.91: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:37,769] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:37,774] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.91:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:37,777] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.91:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:37,778] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:37,925] - [task] [140650554398464] - INFO - server: ip:172.23.121.95 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.121.95', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.121.95', 'uptime': u'28', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.121.95:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3550625792, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:37,926] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2136 [2020-01-22 15:27:37,933] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.121.95:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:37,965] - [remote_util] [140650554398464] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:27:38,226] - [remote_util] [140650554398464] - INFO - Connected to 172.23.121.95 [2020-01-22 15:27:38,524] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:38,960] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.121.95: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:39,025] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:39,029] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.95:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:39,034] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.121.95:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:39,034] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:39,185] - [task] [140650554398464] - INFO - server: ip:172.23.122.62 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.122.62', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.122.62', 'uptime': u'24', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.122.62:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3582611456, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:39,186] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2136 [2020-01-22 15:27:39,192] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.122.62:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:39,222] - [remote_util] [140650554398464] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:27:39,479] - [remote_util] [140650554398464] - INFO - Connected to 172.23.122.62 [2020-01-22 15:27:39,772] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:40,204] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.122.62: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:40,268] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:40,272] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.122.62:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:40,276] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.122.62:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:40,276] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:40,426] - [task] [140650554398464] - INFO - server: ip:172.23.122.73 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.122.73', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.122.73', 'uptime': u'18', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.122.73:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3451981824, 'memoryTotal': 4201684992, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:40,426] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2136 [2020-01-22 15:27:40,433] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.122.73:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:40,465] - [remote_util] [140650554398464] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:27:40,721] - [remote_util] [140650554398464] - INFO - Connected to 172.23.122.73 [2020-01-22 15:27:40,981] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:41,412] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.122.73: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:41,477] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:41,482] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.122.73:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:41,486] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.122.73:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:41,486] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:41,632] - [task] [140650554398464] - INFO - server: ip:172.23.97.220 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'13', 'mcdMemoryReserved': 3230, 'storageTotalRam': 4038, 'hostname': u'172.23.97.220:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.4-3070-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3495538688, 'memoryTotal': 4234354688, 'memoryQuota': 1757, 'mcdMemoryAllocated': 3230, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2020-01-22 15:27:41,632] - [rest_client] [140650554398464] - INFO - pools/default params : memoryQuota=2153 [2020-01-22 15:27:41,638] - [rest_client] [140650554398464] - INFO - settings/web params on 172.23.97.220:8091:username=Administrator&password=password&port=8091 [2020-01-22 15:27:41,669] - [remote_util] [140650554398464] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:27:41,918] - [remote_util] [140650554398464] - INFO - Connected to 172.23.97.220 [2020-01-22 15:27:42,124] - [remote_util] [140650554398464] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:27:42,522] - [remote_util] [140650554398464] - INFO - running command.raw on 172.23.97.220: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:27:42,571] - [remote_util] [140650554398464] - INFO - command executed successfully [2020-01-22 15:27:42,575] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.97.220:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:42,578] - [rest_client] [140650554398464] - INFO - /diag/eval status on 172.23.97.220:8091: True content: [6,0] command: cluster_compat_mode:get_compat_version(). [2020-01-22 15:27:42,579] - [rest_client] [140650554398464] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:27:42,725] - [basetestcase] [140650546005760] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.108.212 **** [2020-01-22 15:27:42,752] - [rest_client] [140650546005760] - ERROR - DELETE http://172.23.108.212:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2020-01-22 15:27:42,753] - [internal_user] [140650546005760] - INFO - Exception while deleting user. Exception is -"User was not found." [2020-01-22 15:27:42,829] - [basetestcase] [140650546005760] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2020-01-22 15:27:42,866] - [basetestcase] [140650546005760] - INFO - done initializing cluster [2020-01-22 15:27:43,709] - [task] [140650554398464] - INFO - adding node 172.23.121.238:8091 to cluster [2020-01-22 15:27:43,710] - [rest_client] [140650554398464] - INFO - adding remote node @172.23.121.238:8091 to this cluster @172.23.108.212:8091 [2020-01-22 15:27:45,823] - [task] [140650554398464] - INFO - adding node 172.23.121.241:8091 to cluster [2020-01-22 15:27:45,823] - [rest_client] [140650554398464] - INFO - adding remote node @172.23.121.241:8091 to this cluster @172.23.108.212:8091 [2020-01-22 15:27:48,019] - [task] [140650554398464] - INFO - adding node 172.23.121.91:8091 to cluster [2020-01-22 15:27:48,019] - [rest_client] [140650554398464] - INFO - adding remote node @172.23.121.91:8091 to this cluster @172.23.108.212:8091 [2020-01-22 15:27:50,225] - [task] [140650554398464] - INFO - adding node 172.23.121.95:8091 to cluster [2020-01-22 15:27:50,225] - [rest_client] [140650554398464] - INFO - adding remote node @172.23.121.95:8091 to this cluster @172.23.108.212:8091 [2020-01-22 15:27:52,655] - [rest_client] [140650554398464] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.108.212,ns_1@172.23.121.238,ns_1@172.23.121.91,ns_1@172.23.121.95,ns_1@172.23.121.241'} [2020-01-22 15:27:52,674] - [rest_client] [140650554398464] - INFO - rebalance operation started [2020-01-22 15:27:52,681] - [rest_client] [140650554398464] - INFO - rebalance percentage : 0.00 % [2020-01-22 15:27:52,681] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 0.00% [2020-01-22 15:28:02,715] - [rest_client] [140650554398464] - INFO - rebalance percentage : 60.00 % [2020-01-22 15:28:02,715] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 60.00% [2020-01-22 15:28:12,746] - [task] [140650554398464] - INFO - Rebalance - status: none, progress: 100.00% [2020-01-22 15:28:12,759] - [task] [140650554398464] - INFO - rebalancing was completed with progress: 100% in 20.0844318867 sec [2020-01-22 15:28:12,761] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:28:13,053] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:28:13,287] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:28:13,752] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-01-22 15:28:13,817] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:28:13,817] - [basetestcase] [140650546005760] - INFO - Enabled diag/eval for non-local hosts from 172.23.108.212 [2020-01-22 15:28:13,931] - [basetestcase] [140650546005760] - INFO - ============== basetestcase setup was finished for test #28 test_killing_eventing_processes_during_eventing_rebalance ============== [2020-01-22 15:28:13,948] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:28:14,252] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:28:14,503] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:28:20,475] - [basetestcase] [140650546005760] - INFO - list of n1ql nodes in cluster: [ip:172.23.121.95 port:8091 ssh_username:root] [2020-01-22 15:28:20,491] - [basetestcase] [140650546005760] - INFO - list of eventing nodes in cluster: [ip:172.23.121.91 port:8091 ssh_username:root, ip:172.23.121.241 port:8091 ssh_username:root] [2020-01-22 15:28:20,498] - [rest_client] [140650546005760] - INFO - settings/indexes params : storageMode=plasma [2020-01-22 15:28:20,503] - [eventing_base] [140650546005760] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2020-01-22 15:28:20,503] - [rest_client] [140650546005760] - INFO - pools/default params : memoryQuota=330 [2020-01-22 15:28:20,509] - [rest_client] [140650546005760] - INFO - pools/default params : indexMemoryQuota=256 [2020-01-22 15:28:20,517] - [rest_client] [140650546005760] - INFO - pools/default params : memoryQuota=700 [2020-01-22 15:28:20,524] - [eventing_rebalance] [140650546005760] - INFO - 100 [2020-01-22 15:28:21,172] - [rest_client] [140650554398464] - INFO - http://172.23.108.212:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 [2020-01-22 15:28:21,181] - [rest_client] [140650554398464] - INFO - 0.01 seconds to create bucket src_bucket [2020-01-22 15:28:21,181] - [bucket_helper] [140650554398464] - INFO - waiting for memcached bucket : src_bucket in 172.23.108.212 to accept set ops [2020-01-22 15:28:22,879] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.108.212:11210 src_bucket [2020-01-22 15:28:22,968] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.121.238:11210 src_bucket [2020-01-22 15:28:23,113] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.108.212:11210 src_bucket [2020-01-22 15:28:23,209] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.121.238:11210 src_bucket [2020-01-22 15:28:23,691] - [task] [140650554398464] - INFO - bucket 'src_bucket' was created with per node RAM quota: 100 [2020-01-22 15:28:24,731] - [rest_client] [140650554398464] - INFO - http://172.23.108.212:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 [2020-01-22 15:28:24,743] - [rest_client] [140650554398464] - INFO - 0.01 seconds to create bucket dst_bucket [2020-01-22 15:28:24,743] - [bucket_helper] [140650554398464] - INFO - waiting for memcached bucket : dst_bucket in 172.23.108.212 to accept set ops [2020-01-22 15:28:26,446] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.108.212:11210 dst_bucket [2020-01-22 15:28:26,553] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.121.238:11210 dst_bucket [2020-01-22 15:28:26,660] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.108.212:11210 dst_bucket [2020-01-22 15:28:27,082] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.121.238:11210 dst_bucket [2020-01-22 15:28:27,588] - [task] [140650554398464] - INFO - bucket 'dst_bucket' was created with per node RAM quota: 100 [2020-01-22 15:28:28,631] - [rest_client] [140650554398464] - INFO - http://172.23.108.212:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=400 [2020-01-22 15:28:28,646] - [rest_client] [140650554398464] - INFO - 0.01 seconds to create bucket metadata [2020-01-22 15:28:28,646] - [bucket_helper] [140650554398464] - INFO - waiting for memcached bucket : metadata in 172.23.108.212 to accept set ops [2020-01-22 15:28:30,327] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.108.212:11210 metadata [2020-01-22 15:28:30,426] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.121.238:11210 metadata [2020-01-22 15:28:30,548] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.108.212:11210 metadata [2020-01-22 15:28:30,645] - [data_helper] [140650554398464] - INFO - creating direct client 172.23.121.238:11210 metadata [2020-01-22 15:28:31,163] - [task] [140650554398464] - INFO - bucket 'metadata' was created with per node RAM quota: 400 [2020-01-22 15:28:31,297] - [basetestcase] [140650546005760] - INFO - list of n1ql nodes in cluster: [ip:172.23.121.95 port:8091 ssh_username:root] [2020-01-22 15:28:31,297] - [tuq_helper] [140650546005760] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2020-01-22 15:28:31,303] - [rest_client] [140650546005760] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2020-01-22 15:28:31,469] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 162.053129ms [2020-01-22 15:28:31,469] - [tuq_helper] [140650546005760] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2020-01-22 15:28:31,475] - [rest_client] [140650546005760] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2020-01-22 15:28:33,937] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 2.45962409s [2020-01-22 15:28:33,938] - [tuq_helper] [140650546005760] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2020-01-22 15:28:33,944] - [rest_client] [140650546005760] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2020-01-22 15:28:33,984] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 37.679678ms [2020-01-22 15:28:33,984] - [tuq_helper] [140650546005760] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2020-01-22 15:28:33,990] - [rest_client] [140650546005760] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2020-01-22 15:28:34,019] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 27.058005ms [2020-01-22 15:28:34,019] - [tuq_helper] [140650546005760] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2020-01-22 15:28:34,025] - [rest_client] [140650546005760] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2020-01-22 15:28:36,494] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 2.466283132s [2020-01-22 15:28:36,494] - [tuq_helper] [140650546005760] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2020-01-22 15:28:36,504] - [rest_client] [140650546005760] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2020-01-22 15:28:36,539] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 32.525652ms [2020-01-22 15:28:36,540] - [tuq_helper] [140650546005760] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2020-01-22 15:28:36,545] - [rest_client] [140650546005760] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2020-01-22 15:28:36,575] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 27.25225ms [2020-01-22 15:28:36,575] - [tuq_helper] [140650546005760] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2020-01-22 15:28:36,580] - [rest_client] [140650546005760] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2020-01-22 15:28:39,063] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 2.480341875s [2020-01-22 15:28:39,064] - [tuq_helper] [140650546005760] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2020-01-22 15:28:39,070] - [rest_client] [140650546005760] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2020-01-22 15:28:39,116] - [tuq_helper] [140650546005760] - INFO - TOTAL ELAPSED TIME: 42.208336ms [2020-01-22 15:28:39,367] - [basetestcase] [140650546005760] - INFO - list of eventing nodes in cluster: [ip:172.23.121.91 port:8091 ssh_username:root, ip:172.23.121.241 port:8091 ssh_username:root] [2020-01-22 15:28:39,368] - [eventing_base] [140650546005760] - INFO - Deploying the following handler code : Function_83692320_test_killing_eventing_processes_during_eventing_rebalance [2020-01-22 15:28:39,368] - [eventing_base] [140650546005760] - INFO - function OnUpdate(doc, meta) { var docID= meta.id; // Adding these extra comments to validate MB-30240 log('Before Inserting document', doc); log('Before Inserting document', doc); log('Before Inserting document', doc); var query = INSERT INTO dst_bucket ( KEY, VALUE ) VALUES ( $docID ,'N1QL op'); // Adding these extra comments to validate MB-30240 log('After Inserting document', doc); log('After Inserting document', doc); log('After Inserting document', doc); } function OnDelete(meta) { var docID = meta.id; // Adding these extra comments to validate MB-30240 log('Deleting document', docID); var query = DELETE FROM dst_bucket where meta().id = $docID; // Adding these extra comments to validate MB-30240 log('Deleting document', docID); } [2020-01-22 15:28:40,595] - [eventing_base] [140650546005760] - INFO - deploy Application : {"code":0,"info":{"status":"Stored function config in metakv","warnings":["Function 'Function_83692320_test_killing_eventing_processes_during_eventing_rebalance' uses Beta features. Do not use in production environments"]}} [2020-01-22 15:28:40,642] - [basetestcase] [140650546005760] - INFO - sleep for 30 secs. Waiting for eventing node to come out of bootstrap state... ... [2020-01-22 15:29:10,813] - [data_helper] [140650546005760] - INFO - creating direct client 172.23.108.212:11210 src_bucket [2020-01-22 15:29:11,250] - [data_helper] [140650546005760] - INFO - creating direct client 172.23.121.238:11210 src_bucket [2020-01-22 15:29:11,366] - [basetestcase] [140650546005760] - INFO - sleep for 15 secs. ... [2020-01-22 15:29:12,217] - [task] [140650554398464] - INFO - adding node 172.23.122.62:8091 to cluster [2020-01-22 15:29:12,218] - [rest_client] [140650554398464] - INFO - adding remote node @172.23.122.62:8091 to this cluster @172.23.108.212:8091 [2020-01-22 15:29:14,919] - [rest_client] [140650554398464] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.121.91,ns_1@172.23.121.238,ns_1@172.23.122.62,ns_1@172.23.121.241,ns_1@172.23.108.212,ns_1@172.23.121.95'} [2020-01-22 15:29:14,931] - [rest_client] [140650554398464] - INFO - rebalance operation started [2020-01-22 15:29:14,939] - [rest_client] [140650554398464] - INFO - rebalance percentage : 0.00 % [2020-01-22 15:29:14,939] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 0.00% [2020-01-22 15:29:24,972] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:29:24,972] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.00% [2020-01-22 15:29:26,389] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:29:29,392] - [rest_client] [140650546005760] - INFO - rebalance reached >50% in 3.00986599922 seconds [2020-01-22 15:29:29,398] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:29:29,692] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:29:30,006] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:29:30,451] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: killall -9 eventing-consumer [2020-01-22 15:29:30,581] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:29:30,704] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:29:31,000] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:29:31,334] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:29:31,841] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: killall -9 eventing-consumer [2020-01-22 15:29:31,973] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:29:32,092] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:29:32,392] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:29:32,666] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:29:33,103] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: killall -9 eventing-producer [2020-01-22 15:29:33,234] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:29:35,008] - [rest_client] [140650554398464] - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2020-01-22 15:29:35,047] - [rest_client] [140650554398464] - INFO - Latest logs from UI on 172.23.108.212: [2020-01-22 15:29:35,048] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {lost_connection,shutdown}}', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:33.219Z', u'module': u'ns_orchestrator', u'tstamp': 1579735773219, u'type': u'critical'} [2020-01-22 15:29:35,049] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.121.241', u'code': 0, u'text': u"Service 'eventing' exited with status 137. Restarting. Messages:\n2020-01-22T15:29:32.757-08:00 [Info] DCPT[eventing:tbahHeMq-22:{eventing:tbahHeMq-21:172.23.121.241:8096_172.23.108.212:11210_worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1}/0] dcp stats bytes: 638002 buffacks: 0 toAckBytes: 633442 streamreqs: 114 snapshots: 863 mutations: 1197 streamends: 0 closestreams: 0\n2020-01-22T15:29:33.001-08:00 [Info] eventing-consumer [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1:34473:118694] Getting credentials afresh for 172.23.121.95:8093\n2020-01-22T15:29:33.013-08:00 [Info] eventing-consumer [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0:42380:118688] Getting credentials afresh for 172.23.121.95:8093\n2020-01-22T15:29:33.017-08:00 [Info] eventing-consumer [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:34808:118707] Getting credentials afresh for 172.23.121.95:8093\n[goport(/opt/couchbase/bin/eventing-producer)] 2020/01/22 15:29:33 child process exited with status 137\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:33.215Z', u'module': u'ns_log', u'tstamp': 1579735773215, u'type': u'info'} [2020-01-22 15:29:35,049] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "src_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:15.981Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735755981, u'type': u'info'} [2020-01-22 15:29:35,049] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket src_bucket', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:15.741Z', u'module': u'ns_rebalancer', u'tstamp': 1579735755741, u'type': u'info'} [2020-01-22 15:29:35,049] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "dst_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:15.680Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735755680, u'type': u'info'} [2020-01-22 15:29:35,050] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket dst_bucket', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:15.423Z', u'module': u'ns_rebalancer', u'tstamp': 1579735755423, u'type': u'info'} [2020-01-22 15:29:35,050] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "metadata" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:15.382Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735755382, u'type': u'info'} [2020-01-22 15:29:35,050] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket metadata', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:15.130Z', u'module': u'ns_rebalancer', u'tstamp': 1579735755130, u'type': u'info'} [2020-01-22 15:29:35,050] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.91','ns_1@172.23.121.238',\n 'ns_1@172.23.122.62','ns_1@172.23.121.241',\n 'ns_1@172.23.108.212','ns_1@172.23.121.95'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:14.925Z', u'module': u'ns_orchestrator', u'tstamp': 1579735754925, u'type': u'info'} [2020-01-22 15:29:35,051] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.122.62', u'code': 0, u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]', u'shortText': u'message', u'serverTime': u'2020-01-22T15:29:14.853Z', u'module': u'memcached_config_mgr', u'tstamp': 1579735754853, u'type': u'info'} [2020-01-22 15:29:35,054] - [eventing_rebalance] [140650546005760] - INFO - Rebalance failed as expected after eventing got killed: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2020-01-22 15:29:35,055] - [basetestcase] [140650546005760] - INFO - sleep for 120 secs. ... [2020-01-22 15:31:35,092] - [basetestcase] [140650546005760] - INFO - sleep for 30 secs. ... [2020-01-22 15:31:35,248] - [rest_client] [140650554398464] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.121.91,ns_1@172.23.121.238,ns_1@172.23.122.62,ns_1@172.23.121.241,ns_1@172.23.108.212,ns_1@172.23.121.95'} [2020-01-22 15:31:35,266] - [rest_client] [140650554398464] - INFO - rebalance operation started [2020-01-22 15:31:35,276] - [rest_client] [140650554398464] - INFO - rebalance percentage : 0.00 % [2020-01-22 15:31:35,277] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 0.00% [2020-01-22 15:31:45,314] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:31:45,314] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.00% [2020-01-22 15:31:55,357] - [rest_client] [140650554398464] - INFO - rebalance percentage : 53.14 % [2020-01-22 15:31:55,358] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 53.14% [2020-01-22 15:32:05,132] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.96 % [2020-01-22 15:32:05,401] - [rest_client] [140650554398464] - INFO - rebalance percentage : 51.96 % [2020-01-22 15:32:05,402] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 51.96% [2020-01-22 15:32:08,145] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:32:11,157] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.26 % [2020-01-22 15:32:14,168] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.88 % [2020-01-22 15:32:15,445] - [rest_client] [140650554398464] - INFO - rebalance percentage : 56.88 % [2020-01-22 15:32:15,445] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 56.88% [2020-01-22 15:32:17,178] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.44 % [2020-01-22 15:32:20,189] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.01 % [2020-01-22 15:32:23,204] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.39 % [2020-01-22 15:32:25,485] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.54 % [2020-01-22 15:32:25,485] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.54% [2020-01-22 15:32:26,238] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.54 % [2020-01-22 15:32:29,248] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.93 % [2020-01-22 15:32:32,260] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.77 % [2020-01-22 15:32:35,269] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.42 % [2020-01-22 15:32:35,527] - [rest_client] [140650554398464] - INFO - rebalance percentage : 51.42 % [2020-01-22 15:32:35,527] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 51.42% [2020-01-22 15:32:38,280] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.50 % [2020-01-22 15:32:41,293] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.98 % [2020-01-22 15:32:44,307] - [rest_client] [140650546005760] - INFO - rebalance percentage : 57.76 % [2020-01-22 15:32:45,566] - [rest_client] [140650554398464] - INFO - rebalance percentage : 57.76 % [2020-01-22 15:32:45,566] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 57.76% [2020-01-22 15:32:47,322] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.46 % [2020-01-22 15:32:50,333] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.79 % [2020-01-22 15:32:53,344] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.54 % [2020-01-22 15:32:55,605] - [rest_client] [140650554398464] - INFO - rebalance percentage : 51.13 % [2020-01-22 15:32:55,605] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 51.13% [2020-01-22 15:32:56,431] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.13 % [2020-01-22 15:32:59,442] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.98 % [2020-01-22 15:33:02,450] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.67 % [2020-01-22 15:33:05,458] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.59 % [2020-01-22 15:33:05,650] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.59 % [2020-01-22 15:33:05,650] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.59% [2020-01-22 15:33:08,469] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.70 % [2020-01-22 15:33:11,479] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.01 % [2020-01-22 15:33:14,564] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.33 % [2020-01-22 15:33:15,689] - [rest_client] [140650554398464] - INFO - rebalance percentage : 51.33 % [2020-01-22 15:33:15,690] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 51.33% [2020-01-22 15:33:17,616] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.39 % [2020-01-22 15:33:20,643] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.57 % [2020-01-22 15:33:23,660] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.42 % [2020-01-22 15:33:25,738] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.88 % [2020-01-22 15:33:25,738] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.88% [2020-01-22 15:33:26,669] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.88 % [2020-01-22 15:33:29,679] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.44 % [2020-01-22 15:33:32,688] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.84 % [2020-01-22 15:33:35,706] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.80 % [2020-01-22 15:33:35,771] - [rest_client] [140650554398464] - INFO - rebalance percentage : 52.80 % [2020-01-22 15:33:35,771] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 52.80% [2020-01-22 15:33:38,719] - [rest_client] [140650546005760] - INFO - rebalance percentage : 53.34 % [2020-01-22 15:33:41,731] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.98 % [2020-01-22 15:33:44,750] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:33:45,815] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:33:45,815] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.00% [2020-01-22 15:33:47,763] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:33:50,819] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.05 % [2020-01-22 15:33:53,849] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.25 % [2020-01-22 15:33:55,853] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.98 % [2020-01-22 15:33:55,853] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.98% [2020-01-22 15:33:56,861] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.98 % [2020-01-22 15:33:59,878] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.59 % [2020-01-22 15:34:02,995] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.74 % [2020-01-22 15:34:05,904] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.29 % [2020-01-22 15:34:05,904] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.29% [2020-01-22 15:34:06,007] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.29 % [2020-01-22 15:34:09,061] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:34:12,112] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.05 % [2020-01-22 15:34:15,129] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.32 % [2020-01-22 15:34:15,948] - [rest_client] [140650554398464] - INFO - rebalance percentage : 56.30 % [2020-01-22 15:34:15,948] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 56.30% [2020-01-22 15:34:18,138] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.30 % [2020-01-22 15:34:21,167] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.49 % [2020-01-22 15:34:24,201] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:34:25,990] - [rest_client] [140650554398464] - INFO - rebalance percentage : 61.62 % [2020-01-22 15:34:25,990] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 61.62% [2020-01-22 15:34:27,276] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.62 % [2020-01-22 15:34:30,301] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.39 % [2020-01-22 15:34:33,512] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.73 % [2020-01-22 15:34:36,033] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.83 % [2020-01-22 15:34:36,033] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.83% [2020-01-22 15:34:36,942] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.83 % [2020-01-22 15:34:39,951] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.29 % [2020-01-22 15:34:42,960] - [rest_client] [140650546005760] - INFO - rebalance percentage : 52.78 % [2020-01-22 15:34:46,004] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.20 % [2020-01-22 15:34:46,070] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.20 % [2020-01-22 15:34:46,070] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.20% [2020-01-22 15:34:49,029] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.44 % [2020-01-22 15:34:52,039] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.59 % [2020-01-22 15:34:55,097] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.10 % [2020-01-22 15:34:56,114] - [rest_client] [140650554398464] - INFO - rebalance percentage : 57.37 % [2020-01-22 15:34:56,114] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 57.37% [2020-01-22 15:34:58,127] - [rest_client] [140650546005760] - INFO - rebalance percentage : 57.37 % [2020-01-22 15:35:01,136] - [rest_client] [140650546005760] - INFO - rebalance percentage : 53.47 % [2020-01-22 15:35:04,163] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.96 % [2020-01-22 15:35:06,159] - [rest_client] [140650554398464] - INFO - rebalance percentage : 57.57 % [2020-01-22 15:35:06,160] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 57.57% [2020-01-22 15:35:07,172] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:35:10,184] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.20 % [2020-01-22 15:35:13,208] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.93 % [2020-01-22 15:35:16,198] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.93 % [2020-01-22 15:35:16,198] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.93% [2020-01-22 15:35:16,236] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.93 % [2020-01-22 15:35:19,253] - [rest_client] [140650546005760] - INFO - rebalance percentage : 66.75 % [2020-01-22 15:35:22,263] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.73 % [2020-01-22 15:35:25,288] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.63 % [2020-01-22 15:35:26,242] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.63 % [2020-01-22 15:35:26,242] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.63% [2020-01-22 15:35:28,309] - [rest_client] [140650546005760] - INFO - rebalance percentage : 66.89 % [2020-01-22 15:35:31,322] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.10 % [2020-01-22 15:35:34,350] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.45 % [2020-01-22 15:35:36,283] - [rest_client] [140650554398464] - INFO - rebalance percentage : 56.45 % [2020-01-22 15:35:36,283] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 56.45% [2020-01-22 15:35:37,583] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.45 % [2020-01-22 15:35:40,604] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.20 % [2020-01-22 15:35:43,614] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:35:46,327] - [rest_client] [140650554398464] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:35:46,327] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 50.00% [2020-01-22 15:35:46,675] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.49 % [2020-01-22 15:35:49,756] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.49 % [2020-01-22 15:35:52,770] - [rest_client] [140650546005760] - INFO - rebalance percentage : 72.22 % [2020-01-22 15:35:55,788] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.57 % [2020-01-22 15:35:56,372] - [rest_client] [140650554398464] - INFO - rebalance percentage : 55.57 % [2020-01-22 15:35:56,372] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 55.57% [2020-01-22 15:35:58,816] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.66 % [2020-01-22 15:36:01,849] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.46 % [2020-01-22 15:36:04,920] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.46 % [2020-01-22 15:36:06,415] - [rest_client] [140650554398464] - INFO - rebalance percentage : 55.08 % [2020-01-22 15:36:06,415] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 55.08% [2020-01-22 15:36:08,181] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.81 % [2020-01-22 15:36:11,201] - [rest_client] [140650546005760] - INFO - rebalance percentage : 57.23 % [2020-01-22 15:36:14,232] - [rest_client] [140650546005760] - INFO - rebalance percentage : 64.84 % [2020-01-22 15:36:16,450] - [rest_client] [140650554398464] - INFO - rebalance percentage : 60.69 % [2020-01-22 15:36:16,451] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 60.69% [2020-01-22 15:36:17,645] - [rest_client] [140650546005760] - INFO - rebalance percentage : 60.69 % [2020-01-22 15:36:20,955] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.43 % [2020-01-22 15:36:23,979] - [rest_client] [140650546005760] - INFO - rebalance percentage : 69.38 % [2020-01-22 15:36:26,485] - [rest_client] [140650554398464] - INFO - rebalance percentage : 65.48 % [2020-01-22 15:36:26,485] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 65.48% [2020-01-22 15:36:27,005] - [rest_client] [140650546005760] - INFO - rebalance percentage : 65.48 % [2020-01-22 15:36:30,024] - [rest_client] [140650546005760] - INFO - rebalance percentage : 72.61 % [2020-01-22 15:36:33,098] - [rest_client] [140650546005760] - INFO - rebalance percentage : 67.24 % [2020-01-22 15:36:36,106] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.23 % [2020-01-22 15:36:36,534] - [rest_client] [140650554398464] - INFO - rebalance percentage : 61.23 % [2020-01-22 15:36:36,534] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 61.23% [2020-01-22 15:36:39,124] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.08 % [2020-01-22 15:36:42,136] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.57 % [2020-01-22 15:36:45,166] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.54 % [2020-01-22 15:36:46,568] - [rest_client] [140650554398464] - INFO - rebalance percentage : 55.52 % [2020-01-22 15:36:46,568] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 55.52% [2020-01-22 15:36:48,185] - [rest_client] [140650546005760] - INFO - rebalance percentage : 55.52 % [2020-01-22 15:36:51,206] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.00 % [2020-01-22 15:36:54,314] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.15 % [2020-01-22 15:36:56,605] - [rest_client] [140650554398464] - INFO - rebalance percentage : 61.13 % [2020-01-22 15:36:56,605] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 61.13% [2020-01-22 15:36:57,337] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.13 % [2020-01-22 15:37:00,789] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.93 % [2020-01-22 15:37:03,813] - [rest_client] [140650546005760] - INFO - rebalance percentage : 50.98 % [2020-01-22 15:37:06,640] - [rest_client] [140650554398464] - INFO - rebalance percentage : 56.59 % [2020-01-22 15:37:06,640] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 56.59% [2020-01-22 15:37:06,835] - [rest_client] [140650546005760] - INFO - rebalance percentage : 56.59 % [2020-01-22 15:37:09,887] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.32 % [2020-01-22 15:37:12,897] - [rest_client] [140650546005760] - INFO - rebalance percentage : 51.32 % [2020-01-22 15:37:16,007] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.43 % [2020-01-22 15:37:16,679] - [rest_client] [140650554398464] - INFO - rebalance percentage : 61.43 % [2020-01-22 15:37:16,679] - [task] [140650554398464] - INFO - Rebalance - status: running, progress: 61.43% [2020-01-22 15:37:19,229] - [rest_client] [140650546005760] - INFO - rebalance percentage : 61.43 % [2020-01-22 15:37:22,440] - [rest_client] [140650546005760] - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2020-01-22 15:37:24,321] - [rest_client] [140650546005760] - INFO - Latest logs from UI on 172.23.121.91: [2020-01-22 15:37:24,321] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.122.62', u'code': 0, u'text': u"Service 'eventing' exited with status 137. Restarting. Messages:\n2020-01-22T15:37:19.283-08:00 [Info] Consumer::getKvNodesFromVbMap [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0:40890:0] Bucket: src_bucket kvNodes: [172.23.108.212:11210 172.23.121.238:11210]\n2020-01-22T15:37:19.275-08:00 [Info] ServiceMgr::GetCurrentTopology rev: service.Revision(nil)\n2020-01-22T15:37:19.356-08:00 [Info] Consumer::SetRebalanceStatus [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:41075:0] Updated isRebalanceOngoing to true\n2020-01-22T15:37:19.506-08:00 [Info] Consumer::gocbConnectMetaBucketCallback [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1:3] Successfully connected to metadata bucket metadata connStr: couchbase://172.23.108.212:11210,172.23.121.238:11210\n2020-01-22T15:37:19.940-08:00 [Info] ServiceMgr::GetCurrentTopology rev: service.Revision{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2}\n2020-01-22T15:37:19.871-08:00 [Info] ServiceMgr::GetTaskList rev: service.Revision{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2}\n[goport(/opt/couchbase/bin/eventing-producer)] 2020/01/22 15:37:22 child process exited with status 137\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:37:22.187Z', u'module': u'ns_log', u'tstamp': 1579736242187, u'type': u'info'} [2020-01-22 15:37:24,321] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {lost_connection,shutdown}}', u'shortText': u'message', u'serverTime': u'2020-01-22T15:37:19.521Z', u'module': u'ns_orchestrator', u'tstamp': 1579736239521, u'type': u'critical'} [2020-01-22 15:37:24,321] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.121.241', u'code': 0, u'text': u"Service 'eventing' exited with status 137. Restarting. Messages:\n2020-01-22T15:37:17.837-08:00 [Info] DCPT[eventing:tbahHeMq-2660:{eventing:tbahHeMq-2659:172.23.121.241:8096_172.23.121.238:11210_worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2}/0] ##abcd ... stopped\n2020-01-22T15:37:17.794-08:00 [Info] Consumer::Stop [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:44391:124119] Closed all dcpfeed handles\n2020-01-22T15:37:17.912-08:00 [Info] Consumer::addToAggChan [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:44391:124119] Closing dcp feed: {eventing:tbahHeMq-2659:172.23.121.241:8096_172.23.121.238:11210_worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2}, count: 0 for bucket: src_bucket\n2020-01-22T15:37:18.029-08:00 [Info] Consumer::processReqStreamMessages [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:44391:124119] Exiting streamReq processing routine\n[goport(/opt/couchbase/bin/eventing-producer)] 2020/01/22 15:37:18 child process exited with status 137\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:37:18.894Z', u'module': u'ns_log', u'tstamp': 1579736238894, u'type': u'info'} [2020-01-22 15:37:24,321] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "src_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:36.093Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735896093, u'type': u'info'} [2020-01-22 15:37:24,322] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket src_bucket', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.883Z', u'module': u'ns_rebalancer', u'tstamp': 1579735895883, u'type': u'info'} [2020-01-22 15:37:24,322] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "dst_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.871Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735895871, u'type': u'info'} [2020-01-22 15:37:24,322] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket dst_bucket', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.662Z', u'module': u'ns_rebalancer', u'tstamp': 1579735895662, u'type': u'info'} [2020-01-22 15:37:24,322] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "metadata" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.634Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735895634, u'type': u'info'} [2020-01-22 15:37:24,322] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket metadata', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.411Z', u'module': u'ns_rebalancer', u'tstamp': 1579735895411, u'type': u'info'} [2020-01-22 15:37:24,322] - [rest_client] [140650546005760] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.91','ns_1@172.23.121.238',\n 'ns_1@172.23.122.62','ns_1@172.23.121.241',\n 'ns_1@172.23.108.212','ns_1@172.23.121.95'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.255Z', u'module': u'ns_orchestrator', u'tstamp': 1579735895255, u'type': u'info'} [2020-01-22 15:37:24,360] - [basetestcase] [140650546005760] - INFO - list of eventing nodes in cluster: [ip:172.23.121.91 port:8091 ssh_username:root, ip:172.23.122.62 port:8091 ssh_username:root, ip:172.23.121.241 port:8091 ssh_username:root] [2020-01-22 15:37:24,443] - [eventing_base] [140650546005760] - INFO - Go routine dumps for Node 172.23.121.91 is goroutine profile: total 166 15 @ 0x43034a 0x440060 0x6a0b6b 0x45dbc1 # 0x6a0b6a net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 10 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x69eae6 0x54612e 0x54627a 0x69f5a5 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x69eae5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546279 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0x69f5a4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 6 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x486346 0x4864b8 0x77e469 0x78bab6 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x486345 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x4864b7 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0x77e468 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0x78bab5 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x1f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1110 6 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x9b5a70 0x9b5b1a 0x9c9732 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x9b5a6f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0x9b5b19 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0x9c9731 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:282 6 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x69eae6 0x54612e 0x546e0c 0x648d54 0x648832 0x648b13 0x696101 0x69604d 0x6a22ac 0x4fffc2 0x4ffccd 0x4ff718 0x75ac12 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x69eae5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546e0b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0x648d53 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0x648831 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0x648b12 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0x696100 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0x69604c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0x6a22ab net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x75ac11 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 6 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0x79e7de 0x45dbc1 # 0x79e7dd github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:297 6 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0x9ca069 0x45dbc1 # 0x9ca068 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 6 @ 0x43034a 0x4303fe 0x441e0b 0x4640a0 0x9b6cc0 0x9b6573 0x9b849e 0x9b9230 0x45dbc1 # 0x441e0a sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x46409f sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0x9b6cbf gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0x9b6572 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0x9b849d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0x9b922f gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 6 @ 0x43034a 0x440060 0x69fb63 0x45dbc1 # 0x69fb62 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 6 @ 0x43034a 0x440060 0x782b9b 0x45dbc1 # 0x782b9a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:185 6 @ 0x43034a 0x440060 0x9c93b2 0x45dbc1 # 0x9c93b1 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:256 6 @ 0x43034a 0x440060 0xa28a63 0x45dbc1 # 0xa28a62 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:278 6 @ 0x43034a 0x440060 0xab43f0 0x45dbc1 # 0xab43ef github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:635 4 @ 0x43034a 0x4303fe 0x40716b 0x406f05 0xa2b0af 0x45dbc1 # 0xa2b0ae github.com/couchbase/eventing/suptree.(*Supervisor).removeService.func1+0x1ae /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:423 4 @ 0x43034a 0x440060 0xa2a353 0xa27a31 0xa2ae69 0x45dbc1 # 0xa2a352 github.com/couchbase/eventing/suptree.(*Supervisor).sendControl+0xf2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:466 # 0xa27a30 github.com/couchbase/eventing/suptree.(*Supervisor).serviceEnded+0x60 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/messages.go:41 # 0xa2ae68 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x78 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:400 4 @ 0x43034a 0x440060 0xae2d6c 0x45dbc1 # 0xae2d6b github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 3 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a72c9 0xae126c 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a72c8 net.(*TCPListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:259 # 0xae126b github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x4b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:524 3 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a72c9 0xae188b 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a72c8 net.(*TCPListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:259 # 0xae188a github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x20a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:543 3 @ 0x43034a 0x440060 0x99e39f 0x45dbc1 # 0x99e39e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 3 @ 0x43034a 0x440060 0xa49f8d 0x45dbc1 # 0xa49f8c github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x20c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x43034a 0x440060 0xa77aca 0x45dbc1 # 0xa77ac9 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x169 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:965 3 @ 0x43034a 0x440060 0xa7b66e 0x45dbc1 # 0xa7b66d github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1174 2 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x4ce0ee 0x4cae8a 0x54612e 0x546e0c 0x547024 0xaaf80a 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x4ce0ed os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x4cae89 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546e0b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0x547023 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0xaaf809 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:96 2 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x4ce0ee 0x4cae8a 0x54612e 0x546e0c 0x547024 0xaafab2 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x4ce0ed os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x4cae89 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546e0b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0x547023 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0xaafab1 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:109 2 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x6871b8 0x54612e 0x54627a 0x68c443 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x6871b7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546279 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0x68c442 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 2 @ 0x43034a 0x440060 0xb08fb7 0xb0d8e2 0xae3034 0xae245b 0xae2853 0xae39f2 0x45b49b 0x4ac569 0x4abae4 0x712ece 0x45dbc1 # 0xb08fb6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:373 # 0xb0d8e1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0xae3033 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0xae245a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0xae2852 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x45b49a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x4ac568 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x4abae3 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0x712ecd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 2 @ 0x43034a 0x440060 0xb08fb7 0xb0e072 0xae2e84 0xae245b 0xae26f3 0xae3922 0x45b49b 0x4ac569 0x4abae4 0x712ece 0x45dbc1 # 0xb08fb6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:373 # 0xb0e071 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0xae2e83 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0xae245a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0xae26f2 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x45b49a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x4ac568 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x4abae3 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0x712ecd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 2 @ 0x471cf5 0x4d0d38 0x4ca71c 0x4c9ccb 0xa3f68c 0xa48daa 0xa2ae53 0x45dbc1 # 0x471cf4 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x4d0d37 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x4ca71b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x4c9cca os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xa3f68b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0xa48da9 github.com/couchbase/eventing/consumer.(*client).Serve+0x1059 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:119 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x412ee2 0x444df6 0x7b4b72 0x45dbc1 # 0x444df5 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0x7b4b71 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x717ffd 0x4fffc2 0x4ffccd 0x4ff718 0x716b7f 0x714847 0x71413a 0x7137a7 0x718569 0x718dcc 0x754620 0x755af5 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x717ffc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x716b7e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0x714846 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0x714139 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0x7137a6 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0x718568 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0x718dcb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x75461f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0x755af4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x717ffd 0x4fffc2 0x4ffccd 0x4ff718 0x716b7f 0x714847 0x71413a 0x7137a7 0x718569 0x718dcc 0xae2bfd 0xb083ce 0xb1c334 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x717ffc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x716b7e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0x714846 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0x714139 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0x7137a6 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0x718568 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0x718dcb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xae2bfc github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0xb083cd github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:276 # 0xb1c333 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0x63 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:77 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a70b9 0x6915ff 0x60c597 0x690395 0x6909ab 0x69129e 0xb1ca44 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a70b8 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0x6915fe net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0x60c596 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0x690394 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0x6909aa net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0x69129d net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0xb1ca43 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x4b3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:239 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a70b9 0x6915ff 0x690395 0x6900e9 0xb1c493 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a70b8 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0x6915fe net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0x690394 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0x6900e8 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0xb1c492 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x142 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:170 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xa91b50 0xa2aed1 0x45dbc1 # 0xa91b4f github.com/couchbase/eventing/consumer.(*Consumer).Stop+0x12ef /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:416 # 0xa2aed0 github.com/couchbase/eventing/suptree.(*Supervisor).removeService.func1.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xb2e513 0xb38c70 0x42fef2 0x45dbc1 # 0xb2e512 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:673 # 0xb38c6f main.main+0x41f goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:110 # 0x42fef1 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xb33ae3 0x45dbc1 # 0xb33ae2 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:78 1 @ 0x43034a 0x4303fe 0x407dc2 0x407abb 0xa0e037 0x45dbc1 # 0xa0e036 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x43034a 0x4303fe 0x440ce4 0x440909 0x467052 0xa8e402 0xa2ae53 0x45dbc1 # 0x440908 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x467051 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0xa8e401 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1851 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:299 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x43034a 0x4303fe 0x44c8a6 0xa0f793 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xa0f792 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x43034a 0x4303fe 0x44c8a6 0xa34795 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xa34794 github.com/couchbase/eventing/timers.(*storeMap).syncRoutine+0x194 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/timers/timer_store.go:660 1 @ 0x43034a 0x4303fe 0x44c8a6 0xaca014 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xaca013 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:113 1 @ 0x43034a 0x4303fe 0x44c8a6 0xad399c 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xad399b github.com/couchbase/eventing/producer.(*Producer).updateStats+0x14b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:763 1 @ 0x43034a 0x440060 0x6a15f7 0x697ff2 0x64c9c5 0x64c69a 0x64dced 0x798911 0x799858 0x79ac30 0xa22fa9 0xa13e80 0xa0a96b 0xa0a735 0xa165fd 0xaae8d2 0xa138fa 0xa6e148 0xa8e154 0xa2ae53 0x45dbc1 # 0x6a15f6 net/http.(*persistConn).roundTrip+0x5a6 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2033 # 0x697ff1 net/http.(*Transport).RoundTrip+0x8f1 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:422 # 0x64c9c4 net/http.send+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/client.go:252 # 0x64c699 net/http.(*Client).send+0xf9 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/client.go:176 # 0x64dcec net/http.(*Client).Do+0x28c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/client.go:615 # 0x798910 github.com/couchbase/eventing/dcp.queryRestAPI+0x1f0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:302 # 0x799857 github.com/couchbase/eventing/dcp.(*Client).parseURLResponse+0x67 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:415 # 0x79ac2f github.com/couchbase/eventing/dcp.(*Client).GetPool+0x28f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:591 # 0xa22fa8 github.com/couchbase/eventing/util.(*ClusterInfoCache).Fetch.func1+0xe58 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:120 # 0xa13e7f github.com/couchbase/eventing/util.(*RetryHelper).Run+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry_helper.go:36 # 0xa0a96a github.com/couchbase/eventing/util.(*ClusterInfoCache).Fetch+0x9a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:202 # 0xa0a734 github.com/couchbase/eventing/util.FetchNewClusterInfoCache+0x124 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:79 # 0xa165fc github.com/couchbase/eventing/util.CurrentEventingNodeAddress+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/util.go:227 # 0xaae8d1 github.com/couchbase/eventing/consumer.glob..func27+0x161 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/cluster_ops.go:23 # 0xa138f9 github.com/couchbase/eventing/util.Retry+0x139 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry.go:65 # 0xa6e147 github.com/couchbase/eventing/consumer.(*Consumer).startDcp+0x3c7 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:453 # 0xa8e153 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x15a3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:273 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x43034a 0x440060 0x6b6b0f 0x45dbc1 # 0x6b6b0e github.com/couchbase/cbauth/cbauthimpl.(*certNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:221 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb393b9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb393b8 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:41 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb394c9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb394c8 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:53 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb395d9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb395d8 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:65 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb396e9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb396e8 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:77 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb397f9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb397f8 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:89 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb39909 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb39908 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:101 1 @ 0x43034a 0x440060 0xa2b051 0x45dbc1 # 0xa2b050 github.com/couchbase/eventing/suptree.(*Supervisor).removeService.func1+0x150 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:417 1 @ 0x43034a 0x440060 0xa4609e 0x45dbc1 # 0xa4609d github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ad /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 1 @ 0x43034a 0x440060 0xa66733 0x45dbc1 # 0xa66732 github.com/couchbase/eventing/consumer.(*Consumer).processEvents+0x462 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:42 1 @ 0x43034a 0x440060 0xa86ab5 0x45dbc1 # 0xa86ab4 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0x114 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:101 1 @ 0x43034a 0x440060 0xacc78e 0xa2ae53 0x45dbc1 # 0xacc78d github.com/couchbase/eventing/producer.(*Producer).Serve+0x11cd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:186 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x43034a 0x440060 0xad3f85 0x45dbc1 # 0xad3f84 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:788 1 @ 0x466055 0x579403 0x581a95 0x580e78 0x5807b5 0x9b4ef0 0x9ac350 0x99a635 0x99a101 0x99987d 0x9d3cbf 0x9ee857 0x9eea38 0xa9de8f 0xa138fa 0xa8cff3 0xa2ae53 0x45dbc1 # 0x466054 sync.(*Mutex).Unlock+0xd4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/mutex.go:175 # 0x579402 context.(*cancelCtx).Done+0x62 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/context/context.go:331 # 0x581a94 net.dialSerial+0xd4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/dial.go:494 # 0x580e77 net.(*Dialer).DialContext+0x677 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/dial.go:397 # 0x5807b4 net.(*Dialer).Dial+0x74 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/dial.go:320 # 0x9b4eef gopkg.in/couchbase/gocbcore%2ev7.dialMemdConn+0xcf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:55 # 0x9ac34f gopkg.in/couchbase/gocbcore%2ev7.(*Agent).dialMemdClient+0xcf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentrouting.go:37 # 0x99a634 gopkg.in/couchbase/gocbcore%2ev7.(*Agent).connect+0x274 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agent.go:690 # 0x99a100 gopkg.in/couchbase/gocbcore%2ev7.createAgent+0x720 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agent.go:655 # 0x99987c gopkg.in/couchbase/gocbcore%2ev7.CreateAgent+0x7c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agent.go:528 # 0x9d3cbe github.com/couchbase/gocb.createBucket+0x2e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/gocb/bucket.go:39 # 0x9ee856 github.com/couchbase/gocb.(*Cluster).openBucket+0x96 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/gocb/cluster.go:282 # 0x9eea37 github.com/couchbase/gocb.(*Cluster).OpenBucket+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/gocb/cluster.go:296 # 0xa9de8e github.com/couchbase/eventing/consumer.glob..func2+0x45e /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/bucket_ops.go:92 # 0xa138f9 github.com/couchbase/eventing/util.Retry+0x139 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry.go:65 # 0xa8cff2 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x442 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:194 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x4a45ca 0x4b1fea 0x4ef311 0x4ee456 0x4eee00 0x4ee456 0x4f0abd 0x4ee4c3 0x4f0abd 0x4ee4c3 0x4ed9c7 0x4ff847 0x798c3b 0x799858 0x79a557 0x79ac67 0xa22fa9 0xa13e80 0xa0a96b 0xa0a735 0xa165fd 0xaae8d2 0xa138fa 0xa6e148 0xa8e154 0xa2ae53 0x45dbc1 # 0x4a45c9 reflect.(*rtype).Elem+0x149 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/type.go:940 # 0x4b1fe9 reflect.MakeSlice+0x99 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:2104 # 0x4ef310 encoding/json.(*decodeState).array+0x6e0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:572 # 0x4ee455 encoding/json.(*decodeState).value+0x265 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:405 # 0x4eedff encoding/json.(*decodeState).array+0x1cf /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:583 # 0x4ee455 encoding/json.(*decodeState).value+0x265 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:405 # 0x4f0abc encoding/json.(*decodeState).object+0x132c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:776 # 0x4ee4c2 encoding/json.(*decodeState).value+0x2d2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:408 # 0x4f0abc encoding/json.(*decodeState).object+0x132c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:776 # 0x4ee4c2 encoding/json.(*decodeState).value+0x2d2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:408 # 0x4ed9c6 encoding/json.(*decodeState).unmarshal+0x1e6 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/decode.go:189 # 0x4ff846 encoding/json.(*Decoder).Decode+0x1a6 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:73 # 0x798c3a github.com/couchbase/eventing/dcp.queryRestAPI+0x51a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:314 # 0x799857 github.com/couchbase/eventing/dcp.(*Client).parseURLResponse+0x67 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:415 # 0x79a556 github.com/couchbase/eventing/dcp.(*Pool).refresh+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:554 # 0x79ac66 github.com/couchbase/eventing/dcp.(*Client).GetPool+0x2c6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/pools.go:595 # 0xa22fa8 github.com/couchbase/eventing/util.(*ClusterInfoCache).Fetch.func1+0xe58 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:120 # 0xa13e7f github.com/couchbase/eventing/util.(*RetryHelper).Run+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry_helper.go:36 # 0xa0a96a github.com/couchbase/eventing/util.(*ClusterInfoCache).Fetch+0x9a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:202 # 0xa0a734 github.com/couchbase/eventing/util.FetchNewClusterInfoCache+0x124 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/cluster_info.go:79 # 0xa165fc github.com/couchbase/eventing/util.CurrentEventingNodeAddress+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/util.go:227 # 0xaae8d1 github.com/couchbase/eventing/consumer.glob..func27+0x161 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/cluster_ops.go:23 # 0xa138f9 github.com/couchbase/eventing/util.Retry+0x139 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/retry.go:65 # 0xa6e147 github.com/couchbase/eventing/consumer.(*Consumer).startDcp+0x3c7 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:453 # 0xa8e153 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x15a3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:273 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x4e63ca 0x67df00 0x67fc73 0x6a0c0a 0x45dbc1 # 0x4e63c9 strings.(*Replacer).Replace+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/strings/replace.go:69 # 0x67deff net/http.Header.WriteSubset+0x1ff /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/header.go:155 # 0x67fc72 net/http.(*Request).write+0x562 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/request.go:577 # 0x6a0c09 net/http.(*persistConn).writeLoop+0x1e9 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1825 1 @ 0x686cb0 0x45dbc1 # 0x686cb0 net/http.(*connReader).backgroundRead+0x0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:667 1 @ 0x7ac198 0x7abfa0 0x7a8ae4 0x7b42ad 0x7b4631 0xb04eed 0xb1d438 0x68d2b4 0x68ef20 0x68ffcc 0x68c2d1 0x45dbc1 # 0x7ac197 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0x7abf9f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0x7a8ae3 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0x7b42ac net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0x7b4630 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0xb04eec github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:2656 # 0xb1d437 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:92 # 0x68d2b3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0x68ef1f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0x68ffcb net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0x68c2d0 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2020-01-22 15:37:24,457] - [eventing_base] [140650546005760] - INFO - Go routine dumps for Node 172.23.122.62 is goroutine profile: total 51 10 @ 0x43034a 0x440060 0x6a0b6b 0x45dbc1 # 0x6a0b6a net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 6 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x69eae6 0x54612e 0x546e0c 0x648d54 0x648832 0x648b13 0x696101 0x69604d 0x6a22ac 0x4fffc2 0x4ffccd 0x4ff718 0x75ac12 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x69eae5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546e0b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0x648d53 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0x648831 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0x648b12 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0x696100 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0x69604c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0x6a22ab net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x75ac11 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 6 @ 0x43034a 0x440060 0x69fb63 0x45dbc1 # 0x69fb62 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 4 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x69eae6 0x54612e 0x54627a 0x69f5a5 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x69eae5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546279 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0x69f5a4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 2 @ 0x43034a 0x440060 0xae2d6c 0x45dbc1 # 0xae2d6b github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x412ee2 0x444df6 0x7b4b72 0x45dbc1 # 0x444df5 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0x7b4b71 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x717ffd 0x4fffc2 0x4ffccd 0x4ff718 0x716b7f 0x714847 0x71413a 0x7137a7 0x718569 0x718dcc 0x754620 0x755af5 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x717ffc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x716b7e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0x714846 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0x714139 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0x7137a6 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0x718568 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0x718dcb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x75461f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0x755af4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x717ffd 0x4fffc2 0x4ffccd 0x4ff718 0x716b7f 0x714847 0x71413a 0x7137a7 0x718569 0x718dcc 0xae2bfd 0xb083ce 0xb1c334 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x717ffc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x716b7e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0x714846 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0x714139 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0x7137a6 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0x718568 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0x718dcb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xae2bfc github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0xb083cd github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:276 # 0xb1c333 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0x63 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:77 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x686d0a 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x686d09 net/http.(*connReader).backgroundRead+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:668 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x6871b8 0x54612e 0x54627a 0x68c443 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x6871b7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546279 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0x68c442 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a70b9 0x6915ff 0x60c597 0x690395 0x6909ab 0x69129e 0xb1ca44 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a70b8 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0x6915fe net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0x60c596 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0x690394 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0x6909aa net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0x69129d net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0xb1ca43 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x4b3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:239 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a70b9 0x6915ff 0x690395 0x6900e9 0xb1c493 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a70b8 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0x6915fe net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0x690394 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0x6900e8 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0xb1c492 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x142 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:170 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xabfb59 0xb299d6 0xb39c4d 0x7584f6 0x7580f8 0x758af5 0xb394c9 0x45dbc1 # 0xabfb58 github.com/couchbase/eventing/producer.(*Producer).SignalBootstrapFinish+0x158 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/exported_functions.go:245 # 0xb299d5 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).SettingsChangeCallback+0xf05 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:357 # 0xb39c4c github.com/couchbase/eventing/supervisor.(*SuperSupervisor).SettingsChangeCallback-fm+0x7c goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:53 # 0x7584f5 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x3b5 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:293 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb394c8 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:53 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xb2e513 0xb38c70 0x42fef2 0x45dbc1 # 0xb2e512 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:673 # 0xb38c6f main.main+0x41f goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:110 # 0x42fef1 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xb33ae3 0x45dbc1 # 0xb33ae2 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:78 1 @ 0x43034a 0x4303fe 0x44c8a6 0xa0f793 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xa0f792 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x43034a 0x4303fe 0x44c8a6 0xa34795 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xa34794 github.com/couchbase/eventing/timers.(*storeMap).syncRoutine+0x194 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/timers/timer_store.go:660 1 @ 0x43034a 0x4303fe 0x44c8a6 0xaba078 0xacb630 0xa2ae53 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xaba077 github.com/couchbase/eventing/producer.(*Producer).parseDepcfg+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/depcfg_parser.go:24 # 0xacb62f github.com/couchbase/eventing/producer.(*Producer).Serve+0x6f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:90 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x43034a 0x440060 0x6b6b0f 0x45dbc1 # 0x6b6b0e github.com/couchbase/cbauth/cbauthimpl.(*certNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:221 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb393b9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb393b8 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:41 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb395d9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb395d8 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:65 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb396e9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb396e8 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:77 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb397f9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb397f8 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:89 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb39909 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb39908 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:101 1 @ 0x43034a 0x440060 0xa28a63 0x45dbc1 # 0xa28a62 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:278 1 @ 0x43034a 0x440060 0xb08fb7 0xb0d8e2 0xae3034 0xae245b 0xae2853 0xae39f2 0x45b49b 0x4ac569 0x4abae4 0x712ece 0x45dbc1 # 0xb08fb6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:373 # 0xb0d8e1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0xae3033 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0xae245a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0xae2852 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x45b49a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x4ac568 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x4abae3 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0x712ecd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x43034a 0x440060 0xb08fb7 0xb0e072 0xae2e84 0xae245b 0xae26f3 0xae3922 0x45b49b 0x4ac569 0x4abae4 0x712ece 0x45dbc1 # 0xb08fb6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:373 # 0xb0e071 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0xae2e83 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0xae245a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0xae26f2 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x45b49a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x4ac568 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x4abae3 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0x712ecd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x7ac198 0x7abfa0 0x7a8ae4 0x7b42ad 0x7b4631 0xb04eed 0xb1d438 0x68d2b4 0x68ef20 0x68ffcc 0x68c2d1 0x45dbc1 # 0x7ac197 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0x7abf9f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0x7a8ae3 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0x7b42ac net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0x7b4630 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0xb04eec github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:2656 # 0xb1d437 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:92 # 0x68d2b3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0x68ef1f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0x68ffcb net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0x68c2d0 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2020-01-22 15:37:24,472] - [eventing_base] [140650546005760] - INFO - Go routine dumps for Node 172.23.121.241 is goroutine profile: total 55 11 @ 0x43034a 0x440060 0x6a0b6b 0x45dbc1 # 0x6a0b6a net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 6 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x69eae6 0x54612e 0x546e0c 0x648d54 0x648832 0x648b13 0x696101 0x69604d 0x6a22ac 0x4fffc2 0x4ffccd 0x4ff718 0x75ac12 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x69eae5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546e0b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0x648d53 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0x648831 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0x648b12 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0x696100 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0x69604c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0x6a22ab net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x75ac11 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 6 @ 0x43034a 0x440060 0x69fb63 0x45dbc1 # 0x69fb62 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 5 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x69eae6 0x54612e 0x54627a 0x69f5a5 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x69eae5 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546279 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0x69f5a4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 2 @ 0x43034a 0x440060 0xae2d6c 0x45dbc1 # 0xae2d6b github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x412ee2 0x444df6 0x7b4b72 0x45dbc1 # 0x444df5 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0x7b4b71 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x717ffd 0x4fffc2 0x4ffccd 0x4ff718 0x716b7f 0x714847 0x71413a 0x7137a7 0x718569 0x718dcc 0x754620 0x755af5 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x717ffc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x716b7e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0x714846 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0x714139 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0x7137a6 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0x718568 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0x718dcb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x75461f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0x755af4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x5467b8 0x717ffd 0x4fffc2 0x4ffccd 0x4ff718 0x716b7f 0x714847 0x71413a 0x7137a7 0x718569 0x718dcc 0xae2bfd 0xb083ce 0xb1c334 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x5467b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0x717ffc github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x4fffc1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x4ffccc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x4ff717 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0x716b7e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0x714846 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0x714139 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0x7137a6 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0x718568 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0x718dcb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xae2bfc github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0xb083cd github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:276 # 0xb1c333 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0x63 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:77 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x686d0a 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x686d09 net/http.(*connReader).backgroundRead+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:668 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c548d 0x58de7f 0x59f15a 0x6871b8 0x54612e 0x54627a 0x68c443 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c548c internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x58de7e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0x59f159 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x6871b7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0x54612d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0x546279 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0x68c442 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a70b9 0x6915ff 0x60c597 0x690395 0x6909ab 0x69129e 0xb1ca44 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a70b8 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0x6915fe net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0x60c596 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0x690394 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0x6909aa net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0x69129d net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0xb1ca43 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x4b3 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:239 1 @ 0x43034a 0x42b66a 0x42ace7 0x4c45ab 0x4c462d 0x4c6a28 0x58e792 0x5a8ade 0x5a70b9 0x6915ff 0x690395 0x6900e9 0xb1c493 0x45dbc1 # 0x42ace6 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x4c45aa internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x4c462c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x4c6a27 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0x58e791 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0x5a8add net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0x5a70b8 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0x6915fe net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0x690394 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0x6900e8 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0xb1c492 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x142 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:170 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xabfb59 0xb299d6 0xb39c4d 0x7584f6 0x7580f8 0x758af5 0xb394c9 0x45dbc1 # 0xabfb58 github.com/couchbase/eventing/producer.(*Producer).SignalBootstrapFinish+0x158 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/exported_functions.go:245 # 0xb299d5 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).SettingsChangeCallback+0xf05 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:357 # 0xb39c4c github.com/couchbase/eventing/supervisor.(*SuperSupervisor).SettingsChangeCallback-fm+0x7c goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:53 # 0x7584f5 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x3b5 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:293 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb394c8 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:53 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xb2e513 0xb38c70 0x42fef2 0x45dbc1 # 0xb2e512 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:673 # 0xb38c6f main.main+0x41f goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:110 # 0x42fef1 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x43034a 0x4303fe 0x407dc2 0x407a7b 0xb33ae3 0x45dbc1 # 0xb33ae2 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:78 1 @ 0x43034a 0x4303fe 0x44c8a6 0xa0f793 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xa0f792 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x43034a 0x4303fe 0x44c8a6 0xa34795 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xa34794 github.com/couchbase/eventing/timers.(*storeMap).syncRoutine+0x194 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/timers/timer_store.go:660 1 @ 0x43034a 0x4303fe 0x44c8a6 0xaca014 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xaca013 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:113 1 @ 0x43034a 0x4303fe 0x44c8a6 0xad46c0 0xacbca2 0xa2ae53 0x45dbc1 # 0x44c8a5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0xad46bf github.com/couchbase/eventing/producer.(*Producer).vbEventingNodeAssign+0x4f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/vbucket_node_assignment.go:21 # 0xacbca1 github.com/couchbase/eventing/producer.(*Producer).Serve+0x6e1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:124 # 0xa2ae52 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:398 1 @ 0x43034a 0x440060 0x6b6b0f 0x45dbc1 # 0x6b6b0e github.com/couchbase/cbauth/cbauthimpl.(*certNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:221 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb393b9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb393b8 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:41 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb395d9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb395d8 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:65 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb396e9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb396e8 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:77 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb397f9 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb397f8 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:89 1 @ 0x43034a 0x440060 0x75843a 0x7580f8 0x758af5 0xb39909 0x45dbc1 # 0x758439 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0x7580f7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0x758af4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0xb39908 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:101 1 @ 0x43034a 0x440060 0xa28a63 0x45dbc1 # 0xa28a62 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:278 1 @ 0x43034a 0x440060 0xad3f85 0x45dbc1 # 0xad3f84 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:788 1 @ 0x43034a 0x440060 0xb08fb7 0xb0d8e2 0xae3034 0xae245b 0xae2853 0xae39f2 0x45b49b 0x4ac569 0x4abae4 0x712ece 0x45dbc1 # 0xb08fb6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:373 # 0xb0d8e1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0xae3033 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0xae245a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0xae2852 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x45b49a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x4ac568 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x4abae3 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0x712ecd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x43034a 0x440060 0xb08fb7 0xb0e072 0xae2e84 0xae245b 0xae26f3 0xae3922 0x45b49b 0x4ac569 0x4abae4 0x712ece 0x45dbc1 # 0xb08fb6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:373 # 0xb0e071 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0xae2e83 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0xae245a github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0xae26f2 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x45b49a runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x4ac568 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x4abae3 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0x712ecd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x7ac198 0x7abfa0 0x7a8ae4 0x7b42ad 0x7b4631 0xb04eed 0xb1d438 0x68d2b4 0x68ef20 0x68ffcc 0x68c2d1 0x45dbc1 # 0x7ac197 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0x7abf9f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0x7a8ae3 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0x7b42ac net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0x7b4630 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0xb04eec github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:2656 # 0xb1d437 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:92 # 0x68d2b3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0x68ef1f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0x68ffcb net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0x68c2d0 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2020-01-22 15:37:24,488] - [basetestcase] [140650546005760] - INFO - list of eventing nodes in cluster: [ip:172.23.121.91 port:8091 ssh_username:root, ip:172.23.122.62 port:8091 ssh_username:root, ip:172.23.121.241 port:8091 ssh_username:root] [2020-01-22 15:37:24,535] - [eventing_base] [140650546005760] - INFO - Stats for Node 172.23.121.91 is [ { "event_processing_stats": { "agg_queue_memory_cap": 132120576, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "is_bootstrapping": 3, "is_rebalance_ongoing": 3, "reb_vb_remaining_to_own": 341, "reb_vb_remaining_to_stream_req": 341, "worker_spawn_counter": 431 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "timestamp": {} }, "failure_stats": { "timestamp": {} }, "function_name": "Function_83692320_test_killing_eventing_processes_during_eventing_rebalance", "gocb_creds_request_counter": 1305, "internal_vb_distribution_stats": { "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0": "[]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1": "[]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2": "[]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 1729, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.121.241:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.121.91:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.121.241:8096": { "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0": "[0-7, 10, 13, 15-113]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1": "[117-124, 128, 130-132, 134-227]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2": "[228, 230-233, 235-246, 248-341]" }, "172.23.121.91:8096": { "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0": "[342-370]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1": "[456-466, 470-569]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2": "[570-574, 577-586, 590-682]" }, "172.23.122.62:8096": { "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0": "[683-696, 698-701, 703-707, 709-737]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1": "[797-806, 809-812, 815-848, 850]", "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2": "[911-917, 919-922, 926-927, 929-963, 965, 968, 971]" } }, "worker_pids": { "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0": 16766, "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1": 16765, "worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2": 0 } } ] [2020-01-22 15:37:24,547] - [eventing_base] [140650546005760] - INFO - Stats for Node 172.23.122.62 is [ { "event_processing_stats": {}, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "timestamp": {} }, "failure_stats": { "timestamp": {} }, "function_name": "Function_83692320_test_killing_eventing_processes_during_eventing_rebalance", "gocb_creds_request_counter": 0, "internal_vb_distribution_stats": {}, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 0, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [], "vb_distribution_stats_from_metadata": {}, "worker_pids": {} } ] [2020-01-22 15:37:24,558] - [eventing_base] [140650546005760] - INFO - Stats for Node 172.23.121.241 is [ { "event_processing_stats": {}, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "timestamp": {} }, "failure_stats": { "timestamp": {} }, "function_name": "Function_83692320_test_killing_eventing_processes_during_eventing_rebalance", "gocb_creds_request_counter": 0, "internal_vb_distribution_stats": {}, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 0, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [], "vb_distribution_stats_from_metadata": {}, "worker_pids": {} } ] [2020-01-22 15:37:24,617] - [basetestcase] [140650546005760] - INFO - list of eventing nodes in cluster: [ip:172.23.121.91 port:8091 ssh_username:root, ip:172.23.122.62 port:8091 ssh_username:root, ip:172.23.121.241 port:8091 ssh_username:root] [2020-01-22 15:37:24,619] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:37:26,716] - [rest_client] [140650554398464] - ERROR - {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed [2020-01-22 15:37:26,754] - [rest_client] [140650554398464] - INFO - Latest logs from UI on 172.23.108.212: [2020-01-22 15:37:26,754] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.122.62', u'code': 0, u'text': u"Service 'eventing' exited with status 137. Restarting. Messages:\n2020-01-22T15:37:19.283-08:00 [Info] Consumer::getKvNodesFromVbMap [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_0:40890:0] Bucket: src_bucket kvNodes: [172.23.108.212:11210 172.23.121.238:11210]\n2020-01-22T15:37:19.275-08:00 [Info] ServiceMgr::GetCurrentTopology rev: service.Revision(nil)\n2020-01-22T15:37:19.356-08:00 [Info] Consumer::SetRebalanceStatus [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:41075:0] Updated isRebalanceOngoing to true\n2020-01-22T15:37:19.506-08:00 [Info] Consumer::gocbConnectMetaBucketCallback [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_1:3] Successfully connected to metadata bucket metadata connStr: couchbase://172.23.108.212:11210,172.23.121.238:11210\n2020-01-22T15:37:19.940-08:00 [Info] ServiceMgr::GetCurrentTopology rev: service.Revision{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2}\n2020-01-22T15:37:19.871-08:00 [Info] ServiceMgr::GetTaskList rev: service.Revision{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x2}\n[goport(/opt/couchbase/bin/eventing-producer)] 2020/01/22 15:37:22 child process exited with status 137\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:37:22.187Z', u'module': u'ns_log', u'tstamp': 1579736242187, u'type': u'info'} [2020-01-22 15:37:26,754] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {lost_connection,shutdown}}', u'shortText': u'message', u'serverTime': u'2020-01-22T15:37:19.521Z', u'module': u'ns_orchestrator', u'tstamp': 1579736239521, u'type': u'critical'} [2020-01-22 15:37:26,754] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.121.241', u'code': 0, u'text': u"Service 'eventing' exited with status 137. Restarting. Messages:\n2020-01-22T15:37:17.837-08:00 [Info] DCPT[eventing:tbahHeMq-2660:{eventing:tbahHeMq-2659:172.23.121.241:8096_172.23.121.238:11210_worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2}/0] ##abcd ... stopped\n2020-01-22T15:37:17.794-08:00 [Info] Consumer::Stop [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:44391:124119] Closed all dcpfeed handles\n2020-01-22T15:37:17.912-08:00 [Info] Consumer::addToAggChan [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:44391:124119] Closing dcp feed: {eventing:tbahHeMq-2659:172.23.121.241:8096_172.23.121.238:11210_worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2}, count: 0 for bucket: src_bucket\n2020-01-22T15:37:18.029-08:00 [Info] Consumer::processReqStreamMessages [worker_Function_83692320_test_killing_eventing_processes_during_eventing_rebalance_2:44391:124119] Exiting streamReq processing routine\n[goport(/opt/couchbase/bin/eventing-producer)] 2020/01/22 15:37:18 child process exited with status 137\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:37:18.894Z', u'module': u'ns_log', u'tstamp': 1579736238894, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "src_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:36.093Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735896093, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket src_bucket', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.883Z', u'module': u'ns_rebalancer', u'tstamp': 1579735895883, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "dst_bucket" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.871Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735895871, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket dst_bucket', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.662Z', u'module': u'ns_rebalancer', u'tstamp': 1579735895662, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Bucket "metadata" rebalance appears to be swap rebalance', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.634Z', u'module': u'ns_vbucket_mover', u'tstamp': 1579735895634, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 0, u'text': u'Started rebalancing bucket metadata', u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.411Z', u'module': u'ns_rebalancer', u'tstamp': 1579735895411, u'type': u'info'} [2020-01-22 15:37:26,755] - [rest_client] [140650554398464] - ERROR - {u'node': u'ns_1@172.23.108.212', u'code': 4, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.121.91','ns_1@172.23.121.238',\n 'ns_1@172.23.122.62','ns_1@172.23.121.241',\n 'ns_1@172.23.108.212','ns_1@172.23.121.95'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes\n", u'shortText': u'message', u'serverTime': u'2020-01-22T15:31:35.255Z', u'module': u'ns_orchestrator', u'tstamp': 1579735895255, u'type': u'info'} [2020-01-22 15:37:41,074] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:38:43,034] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:38:50,056] - [rest_client] [140650546005760] - INFO - /diag/eval status on 172.23.121.91:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-01-22 15:38:50,057] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-01-22 15:38:55,131] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:38:55,132] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-01-22 15:38:55,197] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:38:55,198] - [eventing_base] [140650546005760] - INFO - ===== CORE DUMPS SEEN ON EVENTING NODES, SERVER 172.23.121.91 : 5 crashes seen ===== [2020-01-22 15:38:55,313] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:38:55,673] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:38:56,095] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:38:56,612] - [rest_client] [140650546005760] - INFO - /diag/eval status on 172.23.122.62:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-01-22 15:38:56,613] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-01-22 15:38:59,041] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:38:59,042] - [eventing_base] [140650546005760] - INFO - ===== PANIC OBSERVED IN EVENTING LOGS ON SERVER 172.23.122.62===== [2020-01-22 15:38:59,042] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* [2020-01-22 15:39:01,309] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:01,309] - [eventing_base] [140650546005760] - INFO - ['/opt/couchbase/var/lib/couchbase/logs/eventing.log.4.gz:\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/panic.go:616 +0x81', '/opt/couchbase/var/lib/couchbase/logs/eventing.log.9.gz:\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/panic.go:616 +0x81'] [2020-01-22 15:39:01,309] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-01-22 15:39:01,344] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:01,344] - [eventing_base] [140650546005760] - INFO - ===== CORE DUMPS SEEN ON EVENTING NODES, SERVER 172.23.122.62 : 1 crashes seen ===== [2020-01-22 15:39:01,459] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:39:01,800] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:39:02,252] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:02,709] - [rest_client] [140650546005760] - INFO - /diag/eval status on 172.23.121.241:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-01-22 15:39:02,709] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-01-22 15:39:04,937] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:04,937] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-01-22 15:39:05,003] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:05,003] - [eventing_base] [140650546005760] - INFO - ===== CORE DUMPS SEEN ON EVENTING NODES, SERVER 172.23.121.241 : 1 crashes seen ===== [2020-01-22 15:39:05,119] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:39:05,411] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:39:05,622] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:06,048] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:06,178] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:06,294] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:39:06,550] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:39:06,871] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:07,313] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:07,443] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:07,558] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:39:07,811] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:39:08,105] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:08,524] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:08,653] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:08,769] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:39:09,070] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:39:09,353] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:09,772] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:09,902] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:10,017] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:39:10,277] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:39:10,571] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:11,017] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:11,146] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:11,262] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:39:11,519] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:39:11,835] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:12,252] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:12,382] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:12,497] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:39:12,754] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:39:13,032] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:13,450] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:13,579] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:13,695] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:39:13,939] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:39:14,144] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:14,553] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: kill -9 $(ps aux | pgrep 'memcached') [2020-01-22 15:39:14,650] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:14,766] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:39:15,061] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:39:15,268] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:15,723] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:39:16,008] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:39:16,237] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:16,725] - [remote_util] [140650546005760] - INFO - 172.23.108.212 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:16,800] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:16,800] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: systemctl stop couchbase-server.service [2020-01-22 15:39:26,487] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:26,488] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:39:26,602] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:39:27,521] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:27,522] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:39:27,540] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:27,542] - [remote_util] [140650546005760] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:39:27,833] - [remote_util] [140650546005760] - INFO - Connected to 172.23.108.212 [2020-01-22 15:39:28,064] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:28,567] - [remote_util] [140650546005760] - INFO - 172.23.108.212 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:28,643] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:28,643] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.108.212: systemctl start couchbase-server.service [2020-01-22 15:39:28,710] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:28,710] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:39:28,827] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:39:29,084] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:39:29,407] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:29,956] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:39:30,216] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:39:30,516] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:31,005] - [remote_util] [140650546005760] - INFO - 172.23.121.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:31,063] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:31,064] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: systemctl stop couchbase-server.service [2020-01-22 15:39:34,989] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:34,989] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:39:35,103] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:39:35,571] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:35,571] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:39:35,604] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:35,606] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:39:35,858] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.238 [2020-01-22 15:39:36,184] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:36,690] - [remote_util] [140650546005760] - INFO - 172.23.121.238 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:36,755] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:36,755] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.238: systemctl start couchbase-server.service [2020-01-22 15:39:36,790] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:36,790] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:39:36,905] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:39:37,160] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:39:37,477] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:37,927] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:39:38,184] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:39:38,478] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:38,987] - [remote_util] [140650546005760] - INFO - 172.23.121.241 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:39,043] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:39,043] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: systemctl stop couchbase-server.service [2020-01-22 15:39:42,267] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:42,268] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:39:42,331] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:39:42,350] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:42,350] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:39:42,431] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:42,433] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:39:42,687] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.241 [2020-01-22 15:39:42,981] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:43,482] - [remote_util] [140650546005760] - INFO - 172.23.121.241 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:43,602] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:43,602] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.241: systemctl start couchbase-server.service [2020-01-22 15:39:43,637] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:43,637] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:39:43,753] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:39:44,013] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:39:44,334] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:44,779] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:39:45,063] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:39:45,311] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:45,814] - [remote_util] [140650546005760] - INFO - 172.23.121.91 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:45,872] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:45,873] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: systemctl stop couchbase-server.service [2020-01-22 15:39:48,996] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:48,996] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:39:49,060] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:39:49,079] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:49,079] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:39:49,160] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:49,162] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:39:49,407] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.91 [2020-01-22 15:39:49,689] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:50,193] - [remote_util] [140650546005760] - INFO - 172.23.121.91 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:50,313] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:50,313] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.91: systemctl start couchbase-server.service [2020-01-22 15:39:50,348] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:50,348] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:39:50,463] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:39:50,717] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:39:51,035] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:51,475] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:39:51,729] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:39:52,048] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:52,555] - [remote_util] [140650546005760] - INFO - 172.23.121.95 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:52,677] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:52,677] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: systemctl stop couchbase-server.service [2020-01-22 15:39:55,750] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:55,750] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:39:55,865] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:39:55,883] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:55,884] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:39:55,965] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:55,966] - [remote_util] [140650546005760] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:39:56,224] - [remote_util] [140650546005760] - INFO - Connected to 172.23.121.95 [2020-01-22 15:39:56,522] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:57,028] - [remote_util] [140650546005760] - INFO - 172.23.121.95 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:57,089] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:57,089] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.121.95: systemctl start couchbase-server.service [2020-01-22 15:39:57,124] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:39:57,124] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:39:57,240] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:39:57,497] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:39:57,791] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:58,286] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:39:58,571] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:39:58,856] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:39:59,358] - [remote_util] [140650546005760] - INFO - 172.23.122.62 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:39:59,412] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:39:59,412] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: systemctl stop couchbase-server.service [2020-01-22 15:40:02,535] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:02,535] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:40:02,650] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:40:02,668] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:02,669] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:40:02,749] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:02,751] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:40:03,006] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.62 [2020-01-22 15:40:03,304] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:03,806] - [remote_util] [140650546005760] - INFO - 172.23.122.62 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:40:03,863] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:40:03,863] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.62: systemctl start couchbase-server.service [2020-01-22 15:40:03,898] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:03,898] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:40:04,013] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:40:04,272] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:40:04,550] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:05,041] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:40:05,298] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:40:05,576] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:06,077] - [remote_util] [140650546005760] - INFO - 172.23.122.73 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:40:06,196] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:40:06,197] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: systemctl stop couchbase-server.service [2020-01-22 15:40:08,969] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:08,969] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:40:09,083] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:40:09,101] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:09,102] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:40:09,182] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:09,184] - [remote_util] [140650546005760] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:40:09,440] - [remote_util] [140650546005760] - INFO - Connected to 172.23.122.73 [2020-01-22 15:40:09,698] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:10,198] - [remote_util] [140650546005760] - INFO - 172.23.122.73 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:40:10,316] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:40:10,316] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.122.73: systemctl start couchbase-server.service [2020-01-22 15:40:10,350] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:10,350] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:40:10,466] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:40:10,713] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:40:10,891] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:11,405] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:40:11,641] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:40:11,825] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:12,287] - [remote_util] [140650546005760] - INFO - 172.23.97.220 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:40:12,390] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:40:12,390] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: systemctl stop couchbase-server.service [2020-01-22 15:40:15,162] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:15,162] - [basetestcase] [140650546005760] - INFO - Couchbase stopped [2020-01-22 15:40:15,276] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2020-01-22 15:40:15,295] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:15,295] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2020-01-22 15:40:15,376] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:15,378] - [remote_util] [140650546005760] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:40:15,619] - [remote_util] [140650546005760] - INFO - Connected to 172.23.97.220 [2020-01-22 15:40:15,827] - [remote_util] [140650546005760] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:16,265] - [remote_util] [140650546005760] - INFO - 172.23.97.220 **** The version file /opt/couchbase/ VERSION.txt exists [2020-01-22 15:40:16,369] - [remote_util] [140650546005760] - INFO - Running systemd command on this server [2020-01-22 15:40:16,369] - [remote_util] [140650546005760] - INFO - running command.raw on 172.23.97.220: systemctl start couchbase-server.service [2020-01-22 15:40:16,404] - [remote_util] [140650546005760] - INFO - command executed successfully [2020-01-22 15:40:16,404] - [basetestcase] [140650546005760] - INFO - Couchbase started [2020-01-22 15:40:16,518] - [basetestcase] [140650546005760] - INFO - sleep for 10 secs. ... [2020-01-22 15:40:26,664] - [remote_util] [140650819880768] - INFO - connecting to 172.23.108.212 with username:root [2020-01-22 15:40:26,976] - [remote_util] [140650819880768] - INFO - Connected to 172.23.108.212 [2020-01-22 15:40:27,216] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:40:27,602] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.108.212: /opt/couchbase/bin/cbcollect_info 172.23.108.212-20200122-1540-diag.zip [2020-01-22 15:42:35,506] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:42:37,097] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.108.212: rm -f /root/172.23.108.212-20200122-1540-diag.zip [2020-01-22 15:42:37,171] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:42:37,286] - [remote_util] [140650819880768] - INFO - connecting to 172.23.121.238 with username:root [2020-01-22 15:42:37,542] - [remote_util] [140650819880768] - INFO - Connected to 172.23.121.238 [2020-01-22 15:42:37,861] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:42:38,282] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.238: /opt/couchbase/bin/cbcollect_info 172.23.121.238-20200122-1542-diag.zip [2020-01-22 15:44:08,975] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:44:10,127] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.238: rm -f /root/172.23.121.238-20200122-1542-diag.zip [2020-01-22 15:44:10,163] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:44:10,279] - [remote_util] [140650819880768] - INFO - connecting to 172.23.121.241 with username:root [2020-01-22 15:44:10,535] - [remote_util] [140650819880768] - INFO - Connected to 172.23.121.241 [2020-01-22 15:44:10,855] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:44:11,296] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.241: /opt/couchbase/bin/cbcollect_info 172.23.121.241-20200122-1544-diag.zip [2020-01-22 15:45:44,213] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:45:45,615] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.241: rm -f /root/172.23.121.241-20200122-1544-diag.zip [2020-01-22 15:45:45,652] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:45:45,768] - [remote_util] [140650819880768] - INFO - connecting to 172.23.121.91 with username:root [2020-01-22 15:45:46,026] - [remote_util] [140650819880768] - INFO - Connected to 172.23.121.91 [2020-01-22 15:45:46,321] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:45:46,746] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.91: /opt/couchbase/bin/cbcollect_info 172.23.121.91-20200122-1545-diag.zip [2020-01-22 15:47:42,334] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:47:43,590] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.91: rm -f /root/172.23.121.91-20200122-1545-diag.zip [2020-01-22 15:47:43,688] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:47:43,804] - [remote_util] [140650819880768] - INFO - connecting to 172.23.121.95 with username:root [2020-01-22 15:47:44,067] - [remote_util] [140650819880768] - INFO - Connected to 172.23.121.95 [2020-01-22 15:47:44,384] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:47:44,803] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.95: /opt/couchbase/bin/cbcollect_info 172.23.121.95-20200122-1547-diag.zip [2020-01-22 15:49:18,686] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:49:20,255] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.121.95: rm -f /root/172.23.121.95-20200122-1547-diag.zip [2020-01-22 15:49:20,352] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:49:20,468] - [remote_util] [140650819880768] - INFO - connecting to 172.23.122.62 with username:root [2020-01-22 15:49:20,722] - [remote_util] [140650819880768] - INFO - Connected to 172.23.122.62 [2020-01-22 15:49:21,037] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:49:21,458] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.122.62: /opt/couchbase/bin/cbcollect_info 172.23.122.62-20200122-1549-diag.zip [2020-01-22 15:50:57,112] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:50:58,831] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.122.62: rm -f /root/172.23.122.62-20200122-1549-diag.zip [2020-01-22 15:50:58,929] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:50:59,045] - [remote_util] [140650819880768] - INFO - connecting to 172.23.122.73 with username:root [2020-01-22 15:50:59,292] - [remote_util] [140650819880768] - INFO - Connected to 172.23.122.73 [2020-01-22 15:50:59,573] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:50:59,991] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.122.73: /opt/couchbase/bin/cbcollect_info 172.23.122.73-20200122-1550-diag.zip [2020-01-22 15:52:30,713] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:52:31,942] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.122.73: rm -f /root/172.23.122.73-20200122-1550-diag.zip [2020-01-22 15:52:31,979] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:52:32,094] - [remote_util] [140650819880768] - INFO - connecting to 172.23.97.220 with username:root [2020-01-22 15:52:32,338] - [remote_util] [140650819880768] - INFO - Connected to 172.23.97.220 [2020-01-22 15:52:32,521] - [remote_util] [140650819880768] - INFO - It could be other operating system. Go to check at other location [2020-01-22 15:52:32,882] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.97.220: /opt/couchbase/bin/cbcollect_info 172.23.97.220-20200122-1552-diag.zip [2020-01-22 15:53:45,780] - [remote_util] [140650819880768] - INFO - command executed successfully [2020-01-22 15:53:46,134] - [remote_util] [140650819880768] - INFO - running command.raw on 172.23.97.220: rm -f /root/172.23.97.220-20200122-1552-diag.zip [2020-01-22 15:53:46,216] - [remote_util] [140650819880768] - INFO - command executed successfully