[2019-12-02 07:23:05,470] - [eventing_base] [139937039668992] - INFO - Starting Test: test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance [2019-12-02 07:23:05,475] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:23:11,451] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:23:12,343] - [rest_client] [139937039668992] - INFO - Node version in cluster 6.5.0-4908-enterprise [2019-12-02 07:23:12,356] - [rest_client] [139937039668992] - ERROR - GET http://172.23.105.236:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:23:12,357] - [rest_client] [139937039668992] - INFO - Node versions in cluster [] [2019-12-02 07:23:12,357] - [basetestcase] [139937039668992] - INFO - ============== basetestcase setup was started for test #25 test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance============== [2019-12-02 07:23:12,371] - [rest_client] [139937039668992] - ERROR - GET http://172.23.105.236:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:23:12,573] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.236:8091 [2019-12-02 07:23:12,580] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.236:8091 is running [2019-12-02 07:23:12,588] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.237:8091 [2019-12-02 07:23:12,595] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.237:8091 is running [2019-12-02 07:23:12,603] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.238:8091 [2019-12-02 07:23:12,610] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.238:8091 is running [2019-12-02 07:23:12,618] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.24:8091 [2019-12-02 07:23:12,625] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.24:8091 is running [2019-12-02 07:23:12,633] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.240:8091 [2019-12-02 07:23:12,641] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.240:8091 is running [2019-12-02 07:23:12,649] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.242:8091 [2019-12-02 07:23:12,656] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.242:8091 is running [2019-12-02 07:23:12,664] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.244:8091 [2019-12-02 07:23:12,672] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.244:8091 is running [2019-12-02 07:23:12,679] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.245:8091 [2019-12-02 07:23:12,687] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.245:8091 is running [2019-12-02 07:23:12,688] - [basetestcase] [139937039668992] - INFO - initializing cluster [2019-12-02 07:23:12,691] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:23:19,119] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:23:19,865] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:23:25,679] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:23:26,472] - [remote_util] [139937039668992] - INFO - 172.23.105.236 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:23:26,598] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:23:26,598] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: systemctl stop couchbase-server.service [2019-12-02 07:23:28,269] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:28,270] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:23:28,334] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:23:28,354] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:28,354] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:23:28,436] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:28,440] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:23:34,277] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:23:35,165] - [remote_util] [139937039668992] - INFO - 172.23.105.236 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:23:35,418] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:23:35,419] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: systemctl start couchbase-server.service [2019-12-02 07:23:35,454] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:35,455] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:23:35,572] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:23:41,656] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:23:42,478] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:23:47,926] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:23:48,748] - [remote_util] [139937039668992] - INFO - 172.23.105.237 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:23:48,873] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:23:48,874] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: systemctl stop couchbase-server.service [2019-12-02 07:23:50,445] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:50,445] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:23:50,560] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:23:50,579] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:50,579] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:23:50,661] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:50,665] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:23:56,477] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:23:57,274] - [remote_util] [139937039668992] - INFO - 172.23.105.237 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:23:57,401] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:23:57,401] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: systemctl start couchbase-server.service [2019-12-02 07:23:57,437] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:23:57,437] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:23:57,555] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:24:03,448] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:24:04,214] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:24:10,317] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:24:11,145] - [remote_util] [139937039668992] - INFO - 172.23.105.238 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:24:11,396] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:24:11,397] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: systemctl stop couchbase-server.service [2019-12-02 07:24:12,968] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:12,968] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:24:13,083] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:24:13,102] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:13,102] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:24:13,184] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:13,188] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:24:19,571] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:24:20,409] - [remote_util] [139937039668992] - INFO - 172.23.105.238 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:24:20,535] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:24:20,535] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: systemctl start couchbase-server.service [2019-12-02 07:24:20,571] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:20,571] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:24:20,688] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:24:26,723] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:24:27,542] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:24:33,438] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:24:34,246] - [remote_util] [139937039668992] - INFO - 172.23.105.24 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:24:34,495] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:24:34,495] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: systemctl stop couchbase-server.service [2019-12-02 07:24:36,067] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:36,067] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:24:36,181] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:24:36,201] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:36,201] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:24:36,283] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:36,286] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:24:41,835] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:24:42,679] - [remote_util] [139937039668992] - INFO - 172.23.105.24 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:24:42,804] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:24:42,805] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: systemctl start couchbase-server.service [2019-12-02 07:24:42,840] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:42,841] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:24:42,958] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:24:49,257] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:24:50,057] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:24:55,758] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:24:56,565] - [remote_util] [139937039668992] - INFO - 172.23.105.240 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:24:56,688] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:24:56,689] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: systemctl stop couchbase-server.service [2019-12-02 07:24:58,260] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:58,260] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:24:58,374] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:24:58,393] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:58,394] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:24:58,475] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:24:58,479] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:25:03,932] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:25:04,840] - [remote_util] [139937039668992] - INFO - 172.23.105.240 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:25:05,027] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:25:05,027] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: systemctl start couchbase-server.service [2019-12-02 07:25:05,062] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:05,063] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:25:05,180] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:25:11,151] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:25:11,952] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:25:18,020] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:25:18,828] - [remote_util] [139937039668992] - INFO - 172.23.105.242 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:25:18,953] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:25:18,953] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: systemctl stop couchbase-server.service [2019-12-02 07:25:20,524] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:20,525] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:25:20,639] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:25:20,658] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:20,658] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:25:20,740] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:20,744] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:25:26,359] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:25:27,116] - [remote_util] [139937039668992] - INFO - 172.23.105.242 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:25:27,172] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:25:27,172] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: systemctl start couchbase-server.service [2019-12-02 07:25:27,207] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:27,207] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:25:27,322] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:25:32,405] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:25:33,247] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:25:38,721] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:25:39,567] - [remote_util] [139937039668992] - INFO - 172.23.105.244 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:25:39,756] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:25:39,756] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: systemctl stop couchbase-server.service [2019-12-02 07:25:41,377] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:41,377] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:25:41,441] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:25:41,460] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:41,460] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:25:41,541] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:41,543] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:25:47,183] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:25:48,026] - [remote_util] [139937039668992] - INFO - 172.23.105.244 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:25:48,091] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:25:48,092] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: systemctl start couchbase-server.service [2019-12-02 07:25:48,127] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:25:48,127] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:25:48,243] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:25:53,760] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:25:54,587] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:25:59,742] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:26:00,553] - [remote_util] [139937039668992] - INFO - 172.23.105.245 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:26:00,678] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:26:00,678] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: systemctl stop couchbase-server.service [2019-12-02 07:26:02,349] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:26:02,350] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:26:02,414] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:26:02,433] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:26:02,433] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:26:02,514] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:26:02,516] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:26:07,605] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:26:08,412] - [remote_util] [139937039668992] - INFO - 172.23.105.245 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:26:08,662] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:26:08,662] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: systemctl start couchbase-server.service [2019-12-02 07:26:08,698] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:26:08,699] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:26:08,813] - [basetestcase] [139937039668992] - INFO - sleep for 10 secs. ... [2019-12-02 07:26:19,509] - [task] [139937051461376] - INFO - server: ip:172.23.105.236 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.236', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.236', 'uptime': u'158', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.236:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3606798336, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:19,515] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.236:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:19,515] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:19,525] - [rest_client] [139937051461376] - INFO - /node/controller/setupServices params on 172.23.105.236:8091:services=kv&password=password&user=Administrator&hostname=172.23.105.236 [2019-12-02 07:26:19,534] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.236:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:19,581] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:26:24,694] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.236 [2019-12-02 07:26:25,486] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.236: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:26:25,551] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:26:25,554] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.236:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:25,558] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.236:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:25,558] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:26:25,655] - [task] [139937051461376] - INFO - server: ip:172.23.105.237 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.237', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.237', 'uptime': u'143', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.237:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3625197568, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:25,659] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.237:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:25,659] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:25,666] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.237:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:25,713] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:26:31,292] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.237 [2019-12-02 07:26:32,049] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.237: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:26:32,113] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:26:32,119] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.237:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:32,125] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.237:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:32,125] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:26:32,229] - [task] [139937051461376] - INFO - server: ip:172.23.105.238 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.238', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.238', 'uptime': u'127', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.238:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3674161152, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:32,235] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.238:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:32,235] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:32,244] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.238:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:32,293] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:26:38,325] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.238 [2019-12-02 07:26:39,113] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.238: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:26:39,162] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:26:39,168] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.238:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:39,174] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.238:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:39,175] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:26:39,280] - [task] [139937051461376] - INFO - server: ip:172.23.105.24 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.24', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.24', 'uptime': u'113', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.24:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3655241728, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:39,287] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.24:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:39,287] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:39,296] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.24:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:39,343] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:26:45,177] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.24 [2019-12-02 07:26:45,978] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.24: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:26:46,043] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:26:46,049] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.24:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:46,055] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.24:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:46,055] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:26:46,159] - [task] [139937051461376] - INFO - server: ip:172.23.105.240 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.240', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.240', 'uptime': u'98', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.240:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3640168448, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:46,165] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.240:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:46,166] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:46,175] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.240:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:46,223] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:26:51,310] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.240 [2019-12-02 07:26:52,073] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.240: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:26:52,139] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:26:52,145] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.240:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:52,151] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.240:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:52,151] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:26:52,256] - [task] [139937051461376] - INFO - server: ip:172.23.105.242 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.242', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.242', 'uptime': u'78', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.242:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3655536640, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:52,262] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.242:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:52,263] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:52,271] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.242:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:52,320] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:26:57,919] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.242 [2019-12-02 07:26:58,647] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.242: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:26:58,697] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:26:58,703] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.242:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:58,708] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.242:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:26:58,708] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:26:58,812] - [task] [139937051461376] - INFO - server: ip:172.23.105.244 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.244', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.244', 'uptime': u'68', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.244:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3671519232, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:26:58,818] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.244:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:26:58,818] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:26:58,827] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.244:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:26:58,876] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:27:04,793] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.244 [2019-12-02 07:27:05,560] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.244: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:27:05,610] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:27:05,616] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.244:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:27:05,623] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.244:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:27:05,623] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:27:05,730] - [task] [139937051461376] - INFO - server: ip:172.23.105.245 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.245', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'53', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.245:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4908-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3628625920, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} [2019-12-02 07:27:05,736] - [rest_client] [139937051461376] - ERROR - GET http://172.23.105.245:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password [2019-12-02 07:27:05,737] - [rest_client] [139937051461376] - INFO - pools/default params : memoryQuota=2147 [2019-12-02 07:27:05,746] - [rest_client] [139937051461376] - INFO - settings/web params on 172.23.105.245:8091:username=Administrator&password=password&port=8091 [2019-12-02 07:27:05,795] - [remote_util] [139937051461376] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:27:11,181] - [remote_util] [139937051461376] - INFO - Connected to 172.23.105.245 [2019-12-02 07:27:11,897] - [remote_util] [139937051461376] - INFO - running command.raw on 172.23.105.245: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:27:11,961] - [remote_util] [139937051461376] - INFO - command executed successfully [2019-12-02 07:27:11,965] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.245:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:27:11,968] - [rest_client] [139937051461376] - INFO - /diag/eval status on 172.23.105.245:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). [2019-12-02 07:27:11,968] - [rest_client] [139937051461376] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:27:12,110] - [basetestcase] [139937039668992] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.105.236 **** [2019-12-02 07:27:12,129] - [rest_client] [139937039668992] - ERROR - DELETE http://172.23.105.236:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password [2019-12-02 07:27:12,130] - [internal_user] [139937039668992] - INFO - Exception while deleting user. Exception is -"User was not found." [2019-12-02 07:27:12,184] - [basetestcase] [139937039668992] - INFO - sleep for 5 secs. ... [2019-12-02 07:27:17,190] - [basetestcase] [139937039668992] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2019-12-02 07:27:17,209] - [basetestcase] [139937039668992] - INFO - done initializing cluster [2019-12-02 07:27:18,105] - [task] [139937051461376] - INFO - adding node 172.23.105.237:8091 to cluster [2019-12-02 07:27:18,106] - [rest_client] [139937051461376] - INFO - adding remote node @172.23.105.237:8091 to this cluster @172.23.105.236:8091 [2019-12-02 07:27:20,151] - [task] [139937051461376] - INFO - adding node 172.23.105.238:8091 to cluster [2019-12-02 07:27:20,152] - [rest_client] [139937051461376] - INFO - adding remote node @172.23.105.238:8091 to this cluster @172.23.105.236:8091 [2019-12-02 07:27:22,188] - [task] [139937051461376] - INFO - adding node 172.23.105.24:8091 to cluster [2019-12-02 07:27:22,189] - [rest_client] [139937051461376] - INFO - adding remote node @172.23.105.24:8091 to this cluster @172.23.105.236:8091 [2019-12-02 07:27:24,070] - [task] [139937051461376] - INFO - adding node 172.23.105.240:8091 to cluster [2019-12-02 07:27:24,070] - [rest_client] [139937051461376] - INFO - adding remote node @172.23.105.240:8091 to this cluster @172.23.105.236:8091 [2019-12-02 07:27:25,811] - [rest_client] [139937051461376] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.237,ns_1@172.23.105.240,ns_1@172.23.105.236,ns_1@172.23.105.238,ns_1@172.23.105.24'} [2019-12-02 07:27:25,820] - [rest_client] [139937051461376] - INFO - rebalance operation started [2019-12-02 07:27:25,826] - [rest_client] [139937051461376] - INFO - rebalance percentage : 0.00 % [2019-12-02 07:27:25,826] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 0.00% [2019-12-02 07:27:35,859] - [task] [139937051461376] - INFO - Rebalance - status: none, progress: 100.00% [2019-12-02 07:27:35,869] - [task] [139937051461376] - INFO - rebalancing was completed with progress: 100% in 10.0489010811 sec [2019-12-02 07:27:35,873] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:27:40,919] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:27:41,671] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2019-12-02 07:27:41,737] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:27:41,737] - [basetestcase] [139937039668992] - INFO - Enabled diag/eval for non-local hosts from 172.23.105.236 [2019-12-02 07:27:41,851] - [basetestcase] [139937039668992] - INFO - ============== basetestcase setup was finished for test #25 test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance ============== [2019-12-02 07:27:41,882] - [basetestcase] [139937039668992] - INFO - ------- Cluster statistics ------- [2019-12-02 07:27:41,882] - [basetestcase] [139937039668992] - INFO - 172.23.105.237:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.7481296758104738, 'mem_free': 3564572672, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'kv']} [2019-12-02 07:27:41,883] - [basetestcase] [139937039668992] - INFO - 172.23.105.240:8091 => {'swap_mem_used': 3932160, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3563114496, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'index', u'n1ql']} [2019-12-02 07:27:41,883] - [basetestcase] [139937039668992] - INFO - 172.23.105.236:8091 => {'swap_mem_used': 1048576, 'cpu_utilization': 0.5, 'mem_free': 3546705920, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'kv']} [2019-12-02 07:27:41,883] - [basetestcase] [139937039668992] - INFO - 172.23.105.238:8091 => {'swap_mem_used': 2359296, 'cpu_utilization': 0.2506265664160401, 'mem_free': 3617345536, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'eventing']} [2019-12-02 07:27:41,883] - [basetestcase] [139937039668992] - INFO - 172.23.105.24:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3601375232, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'eventing']} [2019-12-02 07:27:41,884] - [basetestcase] [139937039668992] - INFO - --- End of cluster statistics --- [2019-12-02 07:27:41,887] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:27:47,379] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:27:54,634] - [basetestcase] [139937039668992] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.240 port:8091 ssh_username:root] [2019-12-02 07:27:54,647] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.238 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root] [2019-12-02 07:27:54,653] - [rest_client] [139937039668992] - INFO - settings/indexes params : storageMode=plasma [2019-12-02 07:27:54,659] - [eventing_base] [139937039668992] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2019-12-02 07:27:54,660] - [rest_client] [139937039668992] - INFO - pools/default params : memoryQuota=330 [2019-12-02 07:27:54,665] - [rest_client] [139937039668992] - INFO - pools/default params : indexMemoryQuota=256 [2019-12-02 07:27:54,671] - [rest_client] [139937039668992] - INFO - pools/default params : eventingMemoryQuota=512 [2019-12-02 07:27:54,690] - [basetestcase] [139937039668992] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.240 port:8091 ssh_username:root] [2019-12-02 07:27:54,690] - [rest_client] [139937039668992] - INFO - pools/default params : memoryQuota=700 [2019-12-02 07:27:54,698] - [eventing_rebalance] [139937039668992] - INFO - 100 [2019-12-02 07:27:55,305] - [rest_client] [139937051461376] - INFO - http://172.23.105.236:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 [2019-12-02 07:27:55,313] - [rest_client] [139937051461376] - INFO - 0.01 seconds to create bucket src_bucket [2019-12-02 07:27:55,314] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:27:56,009] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:56,429] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:56,612] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:56,728] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:56,823] - [task] [139937051461376] - WARNING - vbucket map not ready after try 0 [2019-12-02 07:27:56,824] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:27:57,292] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:57,408] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:57,522] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:57,595] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:57,657] - [task] [139937051461376] - WARNING - vbucket map not ready after try 1 [2019-12-02 07:27:57,657] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:27:58,006] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:58,081] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:58,188] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:58,262] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:58,589] - [task] [139937051461376] - WARNING - vbucket map not ready after try 2 [2019-12-02 07:27:58,589] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:27:58,708] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:58,787] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:58,902] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:59,233] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:59,297] - [task] [139937051461376] - WARNING - vbucket map not ready after try 3 [2019-12-02 07:27:59,297] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:27:59,404] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:59,481] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:59,587] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:27:59,914] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:27:59,980] - [task] [139937051461376] - WARNING - vbucket map not ready after try 4 [2019-12-02 07:27:59,980] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : src_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:00,082] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:28:00,154] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:28:00,512] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:28:00,591] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:28:00,657] - [task] [139937051461376] - WARNING - vbucket map not ready after try 5 [2019-12-02 07:28:01,687] - [rest_client] [139937051461376] - INFO - http://172.23.105.236:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 [2019-12-02 07:28:01,695] - [rest_client] [139937051461376] - INFO - 0.01 seconds to create bucket dst_bucket [2019-12-02 07:28:01,695] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:02,353] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:02,779] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:02,949] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:03,064] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:03,165] - [task] [139937051461376] - WARNING - vbucket map not ready after try 0 [2019-12-02 07:28:03,166] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:03,325] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:03,744] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:03,915] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:04,026] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:04,129] - [task] [139937051461376] - WARNING - vbucket map not ready after try 1 [2019-12-02 07:28:04,129] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:04,290] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:04,709] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:04,876] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:04,994] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:05,091] - [task] [139937051461376] - WARNING - vbucket map not ready after try 2 [2019-12-02 07:28:05,091] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:05,558] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:05,670] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:05,843] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:05,964] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:06,060] - [task] [139937051461376] - WARNING - vbucket map not ready after try 3 [2019-12-02 07:28:06,060] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:06,528] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:06,642] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:06,807] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:06,918] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:07,019] - [task] [139937051461376] - WARNING - vbucket map not ready after try 4 [2019-12-02 07:28:07,019] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : dst_bucket in 172.23.105.236 to accept set ops [2019-12-02 07:28:07,487] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:07,600] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:07,765] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 dst_bucket [2019-12-02 07:28:07,880] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:28:08,281] - [task] [139937051461376] - WARNING - vbucket map not ready after try 5 [2019-12-02 07:28:09,325] - [rest_client] [139937051461376] - INFO - http://172.23.105.236:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=400 [2019-12-02 07:28:09,335] - [rest_client] [139937051461376] - INFO - 0.01 seconds to create bucket metadata [2019-12-02 07:28:09,335] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : metadata in 172.23.105.236 to accept set ops [2019-12-02 07:28:10,052] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:10,187] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:10,377] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:10,492] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:10,901] - [task] [139937051461376] - WARNING - vbucket map not ready after try 0 [2019-12-02 07:28:10,902] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : metadata in 172.23.105.236 to accept set ops [2019-12-02 07:28:11,067] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:11,184] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:11,351] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:11,784] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:11,891] - [task] [139937051461376] - WARNING - vbucket map not ready after try 1 [2019-12-02 07:28:11,891] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : metadata in 172.23.105.236 to accept set ops [2019-12-02 07:28:12,051] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:12,163] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:12,327] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:12,749] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:12,848] - [task] [139937051461376] - WARNING - vbucket map not ready after try 2 [2019-12-02 07:28:12,848] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : metadata in 172.23.105.236 to accept set ops [2019-12-02 07:28:13,016] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:13,132] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:13,303] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:13,701] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:13,779] - [task] [139937051461376] - WARNING - vbucket map not ready after try 3 [2019-12-02 07:28:13,779] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : metadata in 172.23.105.236 to accept set ops [2019-12-02 07:28:13,946] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:14,059] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:14,223] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:14,640] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:14,744] - [task] [139937051461376] - WARNING - vbucket map not ready after try 4 [2019-12-02 07:28:14,745] - [bucket_helper] [139937051461376] - INFO - waiting for memcached bucket : metadata in 172.23.105.236 to accept set ops [2019-12-02 07:28:14,908] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:15,029] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:15,498] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.236:11210 metadata [2019-12-02 07:28:15,613] - [data_helper] [139937051461376] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:28:15,710] - [task] [139937051461376] - WARNING - vbucket map not ready after try 5 [2019-12-02 07:28:15,931] - [basetestcase] [139937039668992] - INFO - list of n1ql nodes in cluster: [ip:172.23.105.240 port:8091 ssh_username:root] [2019-12-02 07:28:15,932] - [tuq_helper] [139937039668992] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 07:28:15,941] - [rest_client] [139937039668992] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 07:28:16,182] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 237.530956ms [2019-12-02 07:28:16,182] - [tuq_helper] [139937039668992] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2019-12-02 07:28:16,192] - [rest_client] [139937039668992] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2019-12-02 07:28:18,610] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 2.415059s [2019-12-02 07:28:18,611] - [tuq_helper] [139937039668992] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 07:28:18,620] - [rest_client] [139937039668992] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 07:28:18,658] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 34.294128ms [2019-12-02 07:28:18,659] - [tuq_helper] [139937039668992] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 07:28:18,667] - [rest_client] [139937039668992] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 07:28:18,700] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 30.245584ms [2019-12-02 07:28:18,701] - [tuq_helper] [139937039668992] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2019-12-02 07:28:18,709] - [rest_client] [139937039668992] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2019-12-02 07:28:21,128] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 2.415599194s [2019-12-02 07:28:21,129] - [tuq_helper] [139937039668992] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 07:28:21,138] - [rest_client] [139937039668992] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 07:28:21,174] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 30.888697ms [2019-12-02 07:28:21,175] - [tuq_helper] [139937039668992] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 07:28:21,183] - [rest_client] [139937039668992] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 07:28:21,212] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 26.432257ms [2019-12-02 07:28:21,212] - [tuq_helper] [139937039668992] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2019-12-02 07:28:21,221] - [rest_client] [139937039668992] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2019-12-02 07:28:23,666] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 2.442975576s [2019-12-02 07:28:23,667] - [tuq_helper] [139937039668992] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2019-12-02 07:28:23,675] - [rest_client] [139937039668992] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2019-12-02 07:28:23,715] - [tuq_helper] [139937039668992] - INFO - TOTAL ELAPSED TIME: 37.680253ms [2019-12-02 07:28:24,447] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.238 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root] [2019-12-02 07:28:24,447] - [eventing_base] [139937039668992] - INFO - Deploying the following handler code : Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance with {'buckets': [{'access': 'rw', 'alias': 'dst_bucket', 'bucket_name': 'dst_bucket'}], 'source_bucket': 'src_bucket', 'curl': [], 'metadata_bucket': 'metadata'} [2019-12-02 07:28:24,447] - [eventing_base] [139937039668992] - INFO - function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 300); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 300); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { var docID = context.docID; while (true) { try { var query = DELETE FROM dst_bucket where meta().id = $docID; break; } catch (e) { log(e); } } } function timerCallback(context) { var docID = context.docID; while (true) { try { var query = INSERT INTO dst_bucket ( KEY, VALUE ) VALUES ( $docID ,'timerCallback'); break; } catch (e) { log(e); } } } [2019-12-02 07:28:25,900] - [eventing_base] [139937039668992] - INFO - deploy Application : { "code": 0, "info": { "status": "Stored function: 'Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance' in metakv", "warnings": null } } [2019-12-02 07:28:25,900] - [basetestcase] [139937039668992] - INFO - sleep for 20 secs. Waiting for Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to deployed... ... [2019-12-02 07:28:46,203] - [basetestcase] [139937039668992] - INFO - sleep for 20 secs. Waiting for Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to deployed... ... [2019-12-02 07:29:06,466] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:29:06,556] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:29:06,721] - [basetestcase] [139937039668992] - INFO - sleep for 15 secs. ... [2019-12-02 07:29:06,767] - [task] [139937051461376] - INFO - adding node 172.23.105.242:8091 to cluster [2019-12-02 07:29:06,769] - [rest_client] [139937051461376] - INFO - adding remote node @172.23.105.242:8091 to this cluster @172.23.105.236:8091 [2019-12-02 07:29:09,071] - [rest_client] [139937051461376] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.236,ns_1@172.23.105.240,ns_1@172.23.105.242,ns_1@172.23.105.24,ns_1@172.23.105.237,ns_1@172.23.105.238'} [2019-12-02 07:29:09,081] - [rest_client] [139937051461376] - INFO - rebalance operation started [2019-12-02 07:29:09,093] - [rest_client] [139937051461376] - INFO - rebalance percentage : 0.00 % [2019-12-02 07:29:09,094] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 0.00% [2019-12-02 07:29:19,136] - [rest_client] [139937051461376] - INFO - rebalance percentage : 41.67 % [2019-12-02 07:29:19,136] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 41.67% [2019-12-02 07:29:21,745] - [rest_client] [139937039668992] - INFO - rebalance percentage : 41.67 % [2019-12-02 07:29:24,753] - [rest_client] [139937039668992] - INFO - rebalance reached >41.6666666667% in 3.02968001366 seconds [2019-12-02 07:29:24,762] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:29:30,364] - [rest_client] [139937051461376] - INFO - rebalance percentage : 79.63 % [2019-12-02 07:29:30,365] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 79.63% [2019-12-02 07:29:30,437] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:29:31,209] - [eventing_base] [139937039668992] - INFO - os_info : [2019-12-02 07:29:31,211] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: killall -9 beam.smp [2019-12-02 07:29:31,312] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:29:31,313] - [remote_util] [139937039668992] - INFO - 172.23.105.237:sleep for 2 secs. wait for erlang processes terminated ... [2019-12-02 07:29:31,334] - [data_helper] [139937085032192] - ERROR - Connection reset with error: [Errno 104] Connection reset by peer while _send_op, server is alive? [2019-12-02 07:29:31,454] - [data_helper] [139937085032192] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:29:31,981] - [rest_client] [139937085032192] - ERROR - socket error while connecting to http://172.23.105.237:8091/nodes/self error [Errno 111] Connection refused [2019-12-02 07:29:33,315] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: ps aux | grep beam.smp [2019-12-02 07:29:33,350] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:29:33,475] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:29:33,475] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: systemctl start couchbase-server.service [2019-12-02 07:29:33,509] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:29:33,623] - [basetestcase] [139937039668992] - INFO - sleep for 120 secs. ... [2019-12-02 07:29:34,982] - [rest_client] [139937085032192] - ERROR - socket error while connecting to http://172.23.105.237:8091/nodes/self error [Errno 111] Connection refused [2019-12-02 07:29:37,987] - [rest_client] [139937085032192] - ERROR - socket error while connecting to http://172.23.105.237:8091/nodes/self error [Errno 111] Connection refused [2019-12-02 07:29:40,406] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.18 % [2019-12-02 07:29:40,406] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.18% [2019-12-02 07:29:41,008] - [data_helper] [139937085032192] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:29:50,442] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:29:50,443] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:30:00,476] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:30:00,476] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:30:10,509] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:30:10,509] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:30:20,544] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:30:20,544] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:30:30,576] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:30:30,576] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:30:40,612] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:30:40,612] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:30:50,649] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:30:50,649] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:00,685] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:31:00,685] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:10,724] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:31:10,724] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:20,760] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:31:20,761] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:30,798] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:31:30,799] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:33,713] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.237:8091 [2019-12-02 07:31:33,722] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:31:38,737] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:31:40,835] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:31:40,835] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:43,753] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:31:48,767] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:31:50,872] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:31:50,872] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:31:53,783] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:31:58,799] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:00,908] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:32:00,908] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:32:03,815] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:08,830] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:10,947] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:32:10,947] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:32:13,845] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:18,860] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:20,983] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:32:20,983] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:32:23,875] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:28,891] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:31,019] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:32:31,019] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:32:33,907] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:38,922] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:41,056] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:32:41,056] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:32:43,940] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:48,956] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:51,094] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:32:51,095] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:32:53,972] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:32:58,984] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:01,141] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:33:01,141] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:33:03,998] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:09,013] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:11,176] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:33:11,176] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:33:14,032] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:19,047] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:21,209] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:33:21,210] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:33:24,060] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:29,076] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:31,246] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:33:31,246] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:33:34,091] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:39,107] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:41,282] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:33:41,282] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:33:44,122] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:49,137] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:51,320] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:33:51,320] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:33:54,153] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:33:59,168] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:01,357] - [rest_client] [139937051461376] - INFO - rebalance percentage : 89.28 % [2019-12-02 07:34:01,358] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 89.28% [2019-12-02 07:34:01,406] - [rest_client] [139937051461376] - INFO - Latest logs from UI on 172.23.105.236: [2019-12-02 07:34:01,407] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.236', u'code': 0, u'text': u"Could not auto-failover node ('ns_1@172.23.105.237'). Number of remaining nodes that are running data service is 1. You need at least 2 nodes.", u'shortText': u'message', u'serverTime': u'2019-12-02T07:31:34.817Z', u'module': u'auto_failover', u'tstamp': 1575300694817, u'type': u'info'} [2019-12-02 07:34:01,407] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.237', u'code': 0, u'text': u'Bucket "dst_bucket" loaded on node \'ns_1@172.23.105.237\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-12-02T07:29:40.479Z', u'module': u'ns_memcached', u'tstamp': 1575300580479, u'type': u'info'} [2019-12-02 07:34:01,407] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.237', u'code': 0, u'text': u'Bucket "src_bucket" loaded on node \'ns_1@172.23.105.237\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-12-02T07:29:39.573Z', u'module': u'ns_memcached', u'tstamp': 1575300579573, u'type': u'info'} [2019-12-02 07:34:01,408] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.237', u'code': 0, u'text': u'Bucket "metadata" loaded on node \'ns_1@172.23.105.237\' in 0 seconds.', u'shortText': u'message', u'serverTime': u'2019-12-02T07:29:39.571Z', u'module': u'ns_memcached', u'tstamp': 1575300579571, u'type': u'info'} [2019-12-02 07:34:01,408] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.237', u'code': 1, u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.105.237\'. Version: "6.5.0-4908-enterprise".', u'shortText': u'web start ok', u'serverTime': u'2019-12-02T07:29:38.390Z', u'module': u'menelaus_sup', u'tstamp': 1575300578390, u'type': u'info'} [2019-12-02 07:34:01,408] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.242', u'code': 4, u'text': u"Node 'ns_1@172.23.105.242' saw that node 'ns_1@172.23.105.237' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2019-12-02T07:29:38.257Z', u'module': u'ns_node_disco', u'tstamp': 1575300578257, u'type': u'info'} [2019-12-02 07:34:01,409] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.240', u'code': 4, u'text': u"Node 'ns_1@172.23.105.240' saw that node 'ns_1@172.23.105.237' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2019-12-02T07:29:38.243Z', u'module': u'ns_node_disco', u'tstamp': 1575300578243, u'type': u'info'} [2019-12-02 07:34:01,409] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.24', u'code': 4, u'text': u"Node 'ns_1@172.23.105.24' saw that node 'ns_1@172.23.105.237' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2019-12-02T07:29:38.242Z', u'module': u'ns_node_disco', u'tstamp': 1575300578242, u'type': u'info'} [2019-12-02 07:34:01,410] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.238', u'code': 4, u'text': u"Node 'ns_1@172.23.105.238' saw that node 'ns_1@172.23.105.237' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2019-12-02T07:29:38.233Z', u'module': u'ns_node_disco', u'tstamp': 1575300578233, u'type': u'info'} [2019-12-02 07:34:01,410] - [rest_client] [139937051461376] - ERROR - {u'node': u'ns_1@172.23.105.236', u'code': 4, u'text': u"Node 'ns_1@172.23.105.236' saw that node 'ns_1@172.23.105.237' came up. Tags: []", u'shortText': u'node up', u'serverTime': u'2019-12-02T07:29:38.227Z', u'module': u'ns_node_disco', u'tstamp': 1575300578227, u'type': u'info'} [2019-12-02 07:34:04,186] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:09,201] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:14,216] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:19,231] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:24,246] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:29,260] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:34,276] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:39,291] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:44,307] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:49,322] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:54,337] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:34:59,353] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:04,368] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:09,383] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:14,399] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:19,414] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:24,429] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:29,445] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:34,460] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:39,475] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:44,491] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:49,510] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:54,525] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:35:59,538] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:04,553] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:09,568] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:14,584] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:19,599] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:24,614] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:29,629] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:34,644] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:39,659] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:44,675] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:49,691] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:54,706] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:36:59,722] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:04,737] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:09,750] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:14,765] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:19,774] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:24,791] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:29,807] - [rest_client] [139937039668992] - WARNING - server 172.23.105.237:8091 status is warmup [2019-12-02 07:37:34,813] - [rest_client] [139937039668992] - ERROR - unable to connect to the node 172.23.105.237 even after waiting 360 seconds [2019-12-02 07:37:34,909] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.237:11210 dst_bucket [2019-12-02 07:37:35,077] - [cluster_helper] [139937039668992] - INFO - warmup completed, awesome!!! Warmed up. 0 items [2019-12-02 07:37:35,095] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.237:11210 metadata [2019-12-02 07:37:35,240] - [cluster_helper] [139937039668992] - INFO - warmup completed, awesome!!! Warmed up. 32676 items [2019-12-02 07:37:35,257] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:37:35,737] - [cluster_helper] [139937039668992] - INFO - warmup completed, awesome!!! Warmed up. 18794 items [2019-12-02 07:37:35,747] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:37:41,403] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:37:42,145] - [eventing_base] [139937039668992] - INFO - os_info : [2019-12-02 07:37:42,145] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: killall -9 beam.smp [2019-12-02 07:37:42,242] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:37:42,242] - [remote_util] [139937039668992] - INFO - 172.23.105.238:sleep for 2 secs. wait for erlang processes terminated ... [2019-12-02 07:37:44,245] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: ps aux | grep beam.smp [2019-12-02 07:37:44,283] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:37:44,412] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:37:44,412] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: systemctl start couchbase-server.service [2019-12-02 07:37:44,448] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:37:44,562] - [basetestcase] [139937039668992] - INFO - sleep for 120 secs. ... [2019-12-02 07:39:44,598] - [cluster_helper] [139937039668992] - INFO - waiting for ns_server @ 172.23.105.238:8091 [2019-12-02 07:39:44,604] - [cluster_helper] [139937039668992] - INFO - ns_server @ 172.23.105.238:8091 is running [2019-12-02 07:39:44,605] - [eventing_rebalance] [139937039668992] - INFO - Rebalance failed as expected after erlang got killed: Rebalance Failed: seems like rebalance hangs. please check logs! [2019-12-02 07:39:44,612] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. ... [2019-12-02 07:39:45,406] - [rest_client] [139937051461376] - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.236,ns_1@172.23.105.240,ns_1@172.23.105.242,ns_1@172.23.105.24,ns_1@172.23.105.237,ns_1@172.23.105.238'} [2019-12-02 07:39:45,416] - [rest_client] [139937051461376] - INFO - rebalance operation started [2019-12-02 07:39:45,425] - [rest_client] [139937051461376] - INFO - rebalance percentage : 0.00 % [2019-12-02 07:39:45,426] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 0.00% [2019-12-02 07:39:55,454] - [rest_client] [139937051461376] - INFO - rebalance percentage : 41.67 % [2019-12-02 07:39:55,454] - [task] [139937051461376] - INFO - Rebalance - status: running, progress: 41.67% [2019-12-02 07:40:05,488] - [task] [139937051461376] - INFO - Rebalance - status: none, progress: 100.00% [2019-12-02 07:40:05,508] - [task] [139937051461376] - INFO - rebalancing was completed with progress: 100% in 20.0925529003 sec [2019-12-02 07:40:17,631] - [rest_client] [139937039668992] - INFO - rebalance reached >100% in 3.00976085663 seconds [2019-12-02 07:40:18,627] - [basetestcase] [139937039668992] - INFO - delete 40320 to src_bucket documents... [2019-12-02 07:40:18,708] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.236:11210 src_bucket [2019-12-02 07:40:18,806] - [data_helper] [139937039668992] - INFO - creating direct client 172.23.105.237:11210 src_bucket [2019-12-02 07:40:51,796] - [basetestcase] [139937039668992] - INFO - LOAD IS FINISHED [2019-12-02 07:40:51,906] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.242 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root, ip:172.23.105.238 port:8091 ssh_username:root] [2019-12-02 07:40:51,934] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.242 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root, ip:172.23.105.238 port:8091 ssh_username:root] [2019-12-02 07:40:51,991] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18050 Expected : 0 ... [2019-12-02 07:41:22,057] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18050 Expected : 0 ... [2019-12-02 07:41:52,114] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18050 Expected : 0 ... [2019-12-02 07:42:22,171] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18050 Expected : 0 ... [2019-12-02 07:42:52,232] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18050 Expected : 0 ... [2019-12-02 07:43:22,311] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18554 Expected : 0 ... [2019-12-02 07:43:52,397] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18554 Expected : 0 ... [2019-12-02 07:44:22,487] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18554 Expected : 0 ... [2019-12-02 07:44:52,571] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18554 Expected : 0 ... [2019-12-02 07:45:22,656] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 18554 Expected : 0 ... [2019-12-02 07:45:52,742] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 8347 Expected : 0 ... [2019-12-02 07:46:22,824] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 8347 Expected : 0 ... [2019-12-02 07:46:52,908] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 8100 Expected : 0 ... [2019-12-02 07:47:22,956] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:47:53,019] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:48:23,073] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:48:53,150] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:49:23,229] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:49:53,290] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:50:23,373] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:50:53,446] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:51:23,504] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:51:53,588] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:52:23,641] - [basetestcase] [139937039668992] - INFO - sleep for 30 secs. Waiting for handler code Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance to complete bucket operations... Current : 6261 Expected : 0 ... [2019-12-02 07:52:53,771] - [eventing_base] [139937039668992] - INFO - Documents in source bucket : 0 [2019-12-02 07:52:54,923] - [eventing_base] [139937039668992] - INFO - Stats for Node 172.23.105.242 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 14, "agg_messages_sent_to_worker": 31471, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 6277, "dcp_deletion_sent_to_worker": 6277, "dcp_mutation": 3341, "dcp_mutation_sent_to_worker": 3341, "dcp_snapshot": 7008, "dcp_stream_req_counter": 503, "dcp_stream_req_err_counter": 2, "dcp_streamend": 160, "dcp_streamreq": 495, "execution_stats": 4236, "failure_stats": 4236, "latency_stats": 4236, "lcb_exception_stats": 4236, "log_level": 3, "num_processed_events": 9618, "processed_events_size": 3914220, "thr_count": 3, "thr_map": 3, "timer_events": 12349, "timer_responses_received": 9446, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 6277, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3341, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 6277, "enqueued_dcp_mutation_msg_counter": 3341, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 172, "lcb_retry_failure": 0, "messages_parsed": 31459, "num_processed_events": 9618, "on_delete_failure": 0, "on_delete_success": 6277, "on_update_failure": 0, "on_update_success": 3169, "processed_events_size": 3914220, "timer_create_counter": 9446, "timer_create_failure": 0, "timer_msg_counter": 12349, "timer_responses_sent": 0, "timestamp": { "62135": "2019-12-02T15:52:53Z", "62141": "2019-12-02T15:52:53Z", "62142": "2019-12-02T15:52:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 356, "timeout_count": 12, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "62135": "2019-12-02T15:52:53Z", "62141": "2019-12-02T15:52:53Z", "62142": "2019-12-02T15:52:53Z" }, "v8worker_events_lost": 0 }, "function_id": 1571568087, "function_name": "Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance", "gocb_creds_request_counter": 20, "internal_vb_distribution_stats": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "latency_percentile_stats": { "100": 5355600, "50": 1400, "80": 2100, "90": 2800, "95": 3700, "99": 6900 }, "lcb_creds_request_counter": 132, "lcb_exception_stats": { "12009": 356 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.238:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.105.242:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.105.24:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.105.238:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "172.23.105.242:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "172.23.105.24:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" } }, "worker_pids": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": 62135, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": 62141, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": 62142 } } ] [2019-12-02 07:52:56,837] - [eventing_base] [139937039668992] - INFO - Stats for Node 172.23.105.24 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 38041, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 6257, "dcp_deletion_sent_to_worker": 6257, "dcp_mutation": 7828, "dcp_mutation_sent_to_worker": 7828, "dcp_snapshot": 12585, "dcp_stream_req_counter": 1197, "dcp_stream_req_err_counter": 3, "dcp_streamend": 853, "dcp_streamreq": 1194, "execution_stats": 4377, "failure_stats": 4377, "latency_stats": 4377, "lcb_exception_stats": 4377, "log_level": 3, "num_processed_events": 14085, "processed_events_size": 6927828, "thr_count": 3, "thr_map": 3, "timer_events": 12534, "timer_responses_received": 14084, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 6257, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 7828, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 6257, "enqueued_dcp_mutation_msg_counter": 7828, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 1, "lcb_retry_failure": 0, "messages_parsed": 38029, "num_processed_events": 14085, "on_delete_failure": 0, "on_delete_success": 6257, "on_update_failure": 0, "on_update_success": 7827, "processed_events_size": 6927828, "timer_create_counter": 14084, "timer_create_failure": 0, "timer_msg_counter": 12534, "timer_responses_sent": 0, "timestamp": { "59939": "2019-12-02T15:52:55Z", "59944": "2019-12-02T15:52:55Z", "59946": "2019-12-02T15:52:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 81100, "timeout_count": 45, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "59939": "2019-12-02T15:52:55Z", "59944": "2019-12-02T15:52:55Z", "59946": "2019-12-02T15:52:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1571568087, "function_name": "Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance", "gocb_creds_request_counter": 20, "internal_vb_distribution_stats": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" }, "latency_percentile_stats": { "100": 2501600, "50": 1400, "80": 2200, "90": 2900, "95": 3700, "99": 6400 }, "lcb_creds_request_counter": 152, "lcb_exception_stats": { "12009": 75680, "47": 5420 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.238:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.105.242:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.105.24:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.105.238:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "172.23.105.242:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "172.23.105.24:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" } }, "worker_pids": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": 59939, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": 59944, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": 59946 } } ] [2019-12-02 07:52:58,855] - [eventing_base] [139937039668992] - INFO - Stats for Node 172.23.105.238 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 10, "agg_messages_sent_to_worker": 23238, "agg_queue_memory": 996, "agg_queue_memory_cap": 106954752, "agg_queue_size": 249, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 6263, "dcp_deletion_sent_to_worker": 6263, "dcp_mutation": 3172, "dcp_mutation_sent_to_worker": 3172, "dcp_snapshot": 6601, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 2688, "failure_stats": 2688, "latency_stats": 2688, "lcb_exception_stats": 2688, "log_level": 3, "num_processed_events": 9435, "processed_events_size": 3792040, "thr_count": 3, "thr_map": 3, "timer_events": 87, "timer_responses_received": 9435, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 996, "agg_queue_size": 249, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 6263, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3172, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 6263, "enqueued_dcp_mutation_msg_counter": 3172, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 23226, "num_processed_events": 9435, "on_delete_failure": 0, "on_delete_success": 6263, "on_update_failure": 0, "on_update_success": 3172, "processed_events_size": 3792040, "timer_create_counter": 9435, "timer_create_failure": 0, "timer_msg_counter": 87, "timer_responses_sent": 0, "timestamp": { "130991": "2019-12-02T15:52:57Z", "130992": "2019-12-02T15:52:57Z", "131002": "2019-12-02T15:52:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 318710, "timeout_count": 84, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "130991": "2019-12-02T15:52:57Z", "130992": "2019-12-02T15:52:57Z", "131002": "2019-12-02T15:52:57Z" }, "v8worker_events_lost": 0 }, "function_id": 1571568087, "function_name": "Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "latency_percentile_stats": { "100": 21400, "50": 1300, "80": 2000, "90": 2700, "95": 3500, "99": 6100 }, "lcb_creds_request_counter": 108, "lcb_exception_stats": { "12009": 254663, "47": 64047 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.238:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.105.242:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.105.24:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.105.238:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "172.23.105.242:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "172.23.105.24:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" } }, "worker_pids": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": 130991, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": 130992, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": 131002 } } ] [2019-12-02 07:52:59,629] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.242 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root, ip:172.23.105.238 port:8091 ssh_username:root] [2019-12-02 07:52:59,645] - [eventing_base] [139937039668992] - INFO - Go routine dumps for Node 172.23.105.242 is goroutine profile: total 190 12 @ 0x90f27a 0x91ef90 0xb816ab 0x93cb11 # 0xb816aa net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 9 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa2770c 0xb29894 0xb29372 0xb29653 0xb76c41 0xb76b8d 0xb82dec 0x9e08c2 0x9e05cd 0x9e0018 0xc45bd2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb29893 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb29371 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb29652 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb76c40 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb76b8c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb82deb net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc45bd1 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 9 @ 0x90f27a 0x91ef90 0xb806a3 0x93cb11 # 0xb806a2 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 8 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xce24a0 0xce254a 0xcf6548 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xce249f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0xce2549 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0xcf6547 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf7 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:286 8 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xcf7039 0x93cb11 # 0xcf7038 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 8 @ 0x90f27a 0x90f32e 0x920d3b 0x942ff0 0xce3780 0xce3033 0xce4f5e 0xce5cf0 0x93cb11 # 0x920d3a sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x942fef sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0xce377f gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0xce3032 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0xce4f5d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0xce5cef gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 8 @ 0x90f27a 0x91ef90 0xcf61c2 0x93cb11 # 0xcf61c1 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:257 6 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xc8aeae 0x93cb11 # 0xc8aead github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:307 6 @ 0x90f27a 0x91ef90 0x10e50a0 0x93cb11 # 0x10e509f github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:639 5 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0x965296 0x965408 0xc69679 0xc76c01 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x965295 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x965407 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0xc69678 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0xc76c00 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x220 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1185 5 @ 0x90f27a 0x91ef90 0x1067343 0x93cb11 # 0x1067342 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 5 @ 0x90f27a 0x91ef90 0xc6de6b 0x93cb11 # 0xc6de6a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:190 4 @ 0x90f27a 0x91ef90 0xcca98f 0x93cb11 # 0xcca98e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 4 @ 0x90f27a 0x91ef90 0xd302f4 0x93cb11 # 0xd302f3 gopkg.in/couchbase/gocb%2ev1.(*ThresholdLoggingTracer).loggerRoutine+0x133 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/thresholdlogtracer.go:244 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e023a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e0239 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:102 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e04e2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e04e1 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:115 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094875 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094874 github.com/couchbase/eventing/consumer.(*Consumer).feedbackReadMessageLoop+0x2b4 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:728 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094f9a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094f99 github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop+0x2a9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:792 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa26b7a 0xb800e5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb800e4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x11142b1 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x11142b0 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:599 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x1114c31 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x1114c30 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:633 3 @ 0x90f27a 0x90f32e 0x91fc14 0x91f839 0x945fa2 0x10bebaa 0x1069903 0x93cb11 # 0x91f838 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x945fa1 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0x10beba9 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1c19 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:292 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 3 @ 0x90f27a 0x91ef90 0x1077b8d 0x93cb11 # 0x1077b8c github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ac /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 3 @ 0x90f27a 0x91ef90 0x107bcec 0x93cb11 # 0x107bceb github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x1db /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x90f27a 0x91ef90 0x1092ade 0x93cb11 # 0x1092add github.com/couchbase/eventing/consumer.(*Consumer).sendMessageLoop+0x18d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:561 3 @ 0x90f27a 0x91ef90 0x1098377 0x93cb11 # 0x1098376 github.com/couchbase/eventing/consumer.(*Consumer).processDCPEvents+0x736 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:57 3 @ 0x90f27a 0x91ef90 0x109d9f8 0x93cb11 # 0x109d9f7 github.com/couchbase/eventing/consumer.(*Consumer).processStatsEvents+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:345 3 @ 0x90f27a 0x91ef90 0x109e883 0x93cb11 # 0x109e882 github.com/couchbase/eventing/consumer.(*Consumer).processFilterEvents+0x2e2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:395 3 @ 0x90f27a 0x91ef90 0x10a91fa 0x93cb11 # 0x10a91f9 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x149 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:972 3 @ 0x90f27a 0x91ef90 0x10ad2be 0x93cb11 # 0x10ad2bd github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1197 3 @ 0x90f27a 0x91ef90 0x10b8f75 0x93cb11 # 0x10b8f74 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0x114 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:102 3 @ 0x90f27a 0x91ef90 0x1114485 0x93cb11 # 0x1114484 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:603 3 @ 0x90f27a 0x91ef90 0x1114e05 0x93cb11 # 0x1114e04 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:637 3 @ 0x950c45 0x9b1358 0x9aad3c 0x9aa2eb 0xfb298c 0x107a964 0x1069903 0x93cb11 # 0x950c44 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x9b1357 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x9aad3b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x9aa2ea os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xfb298b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0x107a963 github.com/couchbase/eventing/consumer.(*client).Serve+0x1113 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:125 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 2 @ 0x90f27a 0x91ef90 0x104b591 0x93cb11 # 0x104b590 github.com/couchbase/eventing/util.(*Stats).updater+0x100 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/stats.go:54 2 @ 0x90f27a 0x91ef90 0x11168fc 0x93cb11 # 0x11168fb github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x8f1e12 0x923d26 0xd4d312 0x93cb11 # 0x923d25 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd4d311 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0x111678d 0x114c36e 0x11687c0 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x111678c github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x114c36d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:445 # 0x11687bf github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0xc3f340 0xc409f5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc3f33f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc409f4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb67cf8 0xa26a2e 0xa26b7a 0xb6cf83 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67cf7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb6cf82 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xaecf77 0xb70ed5 0xb714eb 0xb71dde 0x1168f26 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaecf76 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb714ea net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb71ddd net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x1168f25 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:228 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xb70ed5 0xb70c29 0x11689a7 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb70c28 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x11689a6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:186 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x117ce25 0x118820c 0x90ee22 0x93cb11 # 0x117ce24 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x118820b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x90ee21 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x1183093 0x93cb11 # 0x1183092 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:79 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69eb 0x10434c7 0x93cb11 # 0x10434c6 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x1044c23 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x1044c22 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x10fba14 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x10fba13 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:142 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x11066dc 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x11066db github.com/couchbase/eventing/producer.(*Producer).updateStats+0x14b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:873 1 @ 0x90f27a 0x91ef90 0x10fe7e1 0x1069903 0x93cb11 # 0x10fe7e0 github.com/couchbase/eventing/producer.(*Producer).Serve+0x17a0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:198 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x90f27a 0x91ef90 0x1106df5 0x93cb11 # 0x1106df4 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:897 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1151cf2 0x1116bc4 0x1115feb 0x11163e3 0x1117582 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1151cf1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1116bc3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11163e2 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1152482 0x1116a14 0x1115feb 0x1116283 0x11174b2 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1152481 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1116a13 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1116282 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0xb98e91 0x93cb11 # 0xb98e90 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x90f27a 0x91ef90 0xb992ff 0x93cb11 # 0xb992fe github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116924a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169249 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:242 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x11693ca 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x11693c9 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:253 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116954a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169549 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:264 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188939 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188938 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188a49 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188a48 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188b59 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188b58 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188c69 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188c68 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188d79 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188d78 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188e89 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188e88 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0x950bc5 0x94e60f 0x94ca99 0x9a5a48 0xa6e85f 0xa7fb3a 0xb6784a 0x93cb11 # 0x950bc4 syscall.Syscall+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:18 # 0x94e60e syscall.read+0x5e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/zsyscall_linux_amd64.go:749 # 0x94ca98 syscall.Read+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/syscall_unix.go:162 # 0x9a5a47 internal/poll.(*FD).Read+0x117 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:153 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67849 net/http.(*connReader).backgroundRead+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:668 1 @ 0xd44938 0xd44740 0xd41284 0xd4ca4d 0xd4cdd1 0x11463ad 0x1169fb8 0xb6ddf4 0xb6fa60 0xb70b0c 0xb6ce11 0x93cb11 # 0xd44937 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd4473f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd41283 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd4ca4c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd4cdd0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11463ac github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3578 # 0x1169fb7 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb6ddf3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb6fa5f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb70b0b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb6ce10 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2019-12-02 07:52:59,660] - [eventing_base] [139937039668992] - INFO - Go routine dumps for Node 172.23.105.24 is goroutine profile: total 184 14 @ 0x90f27a 0x91ef90 0xb816ab 0x93cb11 # 0xb816aa net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 9 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa2770c 0xb29894 0xb29372 0xb29653 0xb76c41 0xb76b8d 0xb82dec 0x9e08c2 0x9e05cd 0x9e0018 0xc45bd2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb29893 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb29371 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb29652 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb76c40 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb76b8c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb82deb net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc45bd1 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 9 @ 0x90f27a 0x91ef90 0xb806a3 0x93cb11 # 0xb806a2 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 8 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xce24a0 0xce254a 0xcf6548 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xce249f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0xce2549 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0xcf6547 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf7 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:286 8 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xcf7039 0x93cb11 # 0xcf7038 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 8 @ 0x90f27a 0x90f32e 0x920d3b 0x942ff0 0xce3780 0xce3033 0xce4f5e 0xce5cf0 0x93cb11 # 0x920d3a sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x942fef sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0xce377f gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0xce3032 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0xce4f5d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0xce5cef gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 8 @ 0x90f27a 0x91ef90 0xcf61c2 0x93cb11 # 0xcf61c1 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:257 5 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa26b7a 0xb800e5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb800e4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 5 @ 0x90f27a 0x91ef90 0x1067343 0x93cb11 # 0x1067342 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 4 @ 0x90f27a 0x91ef90 0xcca98f 0x93cb11 # 0xcca98e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 4 @ 0x90f27a 0x91ef90 0xd302f4 0x93cb11 # 0xd302f3 gopkg.in/couchbase/gocb%2ev1.(*ThresholdLoggingTracer).loggerRoutine+0x133 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/thresholdlogtracer.go:244 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e023a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e0239 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:102 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e04e2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e04e1 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:115 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0x965296 0x965408 0xc69679 0xc76c01 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x965295 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x965407 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0xc69678 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0xc76c00 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x220 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1185 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094875 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094874 github.com/couchbase/eventing/consumer.(*Consumer).feedbackReadMessageLoop+0x2b4 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:728 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094f9a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094f99 github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop+0x2a9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:792 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x11142b1 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x11142b0 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:599 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x1114c31 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x1114c30 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:633 3 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xc8aeae 0x93cb11 # 0xc8aead github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:307 3 @ 0x90f27a 0x90f32e 0x91fc14 0x91f839 0x945fa2 0x10bebaa 0x1069903 0x93cb11 # 0x91f838 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x945fa1 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0x10beba9 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1c19 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:292 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 3 @ 0x90f27a 0x91ef90 0x1077b8d 0x93cb11 # 0x1077b8c github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ac /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 3 @ 0x90f27a 0x91ef90 0x107bcec 0x93cb11 # 0x107bceb github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x1db /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x90f27a 0x91ef90 0x1092ade 0x93cb11 # 0x1092add github.com/couchbase/eventing/consumer.(*Consumer).sendMessageLoop+0x18d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:561 3 @ 0x90f27a 0x91ef90 0x1098377 0x93cb11 # 0x1098376 github.com/couchbase/eventing/consumer.(*Consumer).processDCPEvents+0x736 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:57 3 @ 0x90f27a 0x91ef90 0x109d9f8 0x93cb11 # 0x109d9f7 github.com/couchbase/eventing/consumer.(*Consumer).processStatsEvents+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:345 3 @ 0x90f27a 0x91ef90 0x109e883 0x93cb11 # 0x109e882 github.com/couchbase/eventing/consumer.(*Consumer).processFilterEvents+0x2e2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:395 3 @ 0x90f27a 0x91ef90 0x10a91fa 0x93cb11 # 0x10a91f9 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x149 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:972 3 @ 0x90f27a 0x91ef90 0x10ad2be 0x93cb11 # 0x10ad2bd github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1197 3 @ 0x90f27a 0x91ef90 0x10b8f75 0x93cb11 # 0x10b8f74 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0x114 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:102 3 @ 0x90f27a 0x91ef90 0x10e50a0 0x93cb11 # 0x10e509f github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:639 3 @ 0x90f27a 0x91ef90 0x1114485 0x93cb11 # 0x1114484 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:603 3 @ 0x90f27a 0x91ef90 0x1114e05 0x93cb11 # 0x1114e04 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:637 3 @ 0x90f27a 0x91ef90 0xc6de6b 0x93cb11 # 0xc6de6a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:190 3 @ 0x950c45 0x9b1358 0x9aad3c 0x9aa2eb 0xfb298c 0x107a964 0x1069903 0x93cb11 # 0x950c44 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x9b1357 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x9aad3b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x9aa2ea os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xfb298b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0x107a963 github.com/couchbase/eventing/consumer.(*client).Serve+0x1113 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:125 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 2 @ 0x90f27a 0x91ef90 0x104b591 0x93cb11 # 0x104b590 github.com/couchbase/eventing/util.(*Stats).updater+0x100 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/stats.go:54 2 @ 0x90f27a 0x91ef90 0x11168fc 0x93cb11 # 0x11168fb github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x8f1e12 0x923d26 0xd4d312 0x93cb11 # 0x923d25 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd4d311 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0x111678d 0x114c36e 0x11687c0 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x111678c github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x114c36d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:445 # 0x11687bf github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0xc3f340 0xc409f5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc3f33f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc409f4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb67cf8 0xa26a2e 0xa26b7a 0xb6cf83 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67cf7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb6cf82 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xaecf77 0xb70ed5 0xb714eb 0xb71dde 0x1168f26 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaecf76 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb714ea net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb71ddd net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x1168f25 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:228 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xb70ed5 0xb70c29 0x11689a7 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb70c28 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x11689a6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:186 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x117ce25 0x118820c 0x90ee22 0x93cb11 # 0x117ce24 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x118820b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x90ee21 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x1183093 0x93cb11 # 0x1183092 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:79 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69eb 0x10434c7 0x93cb11 # 0x10434c6 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x1044c23 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x1044c22 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x10fba14 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x10fba13 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:142 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x11066dc 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x11066db github.com/couchbase/eventing/producer.(*Producer).updateStats+0x14b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:873 1 @ 0x90f27a 0x91ef90 0x10fe7e1 0x1069903 0x93cb11 # 0x10fe7e0 github.com/couchbase/eventing/producer.(*Producer).Serve+0x17a0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:198 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x90f27a 0x91ef90 0x1106df5 0x93cb11 # 0x1106df4 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:897 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1151cf2 0x1116bc4 0x1115feb 0x11163e3 0x1117582 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1151cf1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1116bc3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11163e2 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1152482 0x1116a14 0x1115feb 0x1116283 0x11174b2 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1152481 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1116a13 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1116282 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0xb98e91 0x93cb11 # 0xb98e90 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x90f27a 0x91ef90 0xb992ff 0x93cb11 # 0xb992fe github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116924a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169249 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:242 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x11693ca 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x11693c9 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:253 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116954a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169549 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:264 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188939 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188938 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188a49 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188a48 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188b59 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188b58 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188c69 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188c68 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188d79 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188d78 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188e89 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188e88 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0xb677f0 0x93cb11 # 0xb677f0 net/http.(*connReader).backgroundRead+0x0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:667 1 @ 0xd44938 0xd44740 0xd41284 0xd4ca4d 0xd4cdd1 0x11463ad 0x1169fb8 0xb6ddf4 0xb6fa60 0xb70b0c 0xb6ce11 0x93cb11 # 0xd44937 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd4473f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd41283 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd4ca4c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd4cdd0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11463ac github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3578 # 0x1169fb7 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb6ddf3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb6fa5f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb70b0b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb6ce10 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2019-12-02 07:52:59,679] - [eventing_base] [139937039668992] - INFO - Go routine dumps for Node 172.23.105.238 is goroutine profile: total 196 14 @ 0x90f27a 0x91ef90 0xb816ab 0x93cb11 # 0xb816aa net/http.(*persistConn).writeLoop+0x14a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1822 9 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa2770c 0xb29894 0xb29372 0xb29653 0xb76c41 0xb76b8d 0xb82dec 0x9e08c2 0x9e05cd 0x9e0018 0xc45bd2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xb29893 net/http/internal.readChunkLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:122 # 0xb29371 net/http/internal.(*chunkedReader).beginChunk+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:48 # 0xb29652 net/http/internal.(*chunkedReader).Read+0x112 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/internal/chunked.go:93 # 0xb76c40 net/http.(*body).readLocked+0x60 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:778 # 0xb76b8c net/http.(*body).Read+0xdc /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transfer.go:770 # 0xb82deb net/http.(*bodyEOFSignal).Read+0xdb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:2187 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xc45bd1 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1+0x121 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:270 9 @ 0x90f27a 0x91ef90 0xb806a3 0x93cb11 # 0xb806a2 net/http.(*persistConn).readLoop+0x742 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1717 8 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xce24a0 0xce254a 0xcf6548 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xce249f gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).readFullBuffer+0x8f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:143 # 0xce2549 gopkg.in/couchbase/gocbcore%2ev7.(*memdTcpConn).ReadPacket+0x49 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdconn.go:159 # 0xcf6547 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func2+0xf7 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:286 8 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xcf7039 0x93cb11 # 0xcf7038 gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop.func1+0xf8 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:63 8 @ 0x90f27a 0x90f32e 0x920d3b 0x942ff0 0xce3780 0xce3033 0xce4f5e 0xce5cf0 0x93cb11 # 0x920d3a sync.runtime_notifyListWait+0x10a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:510 # 0x942fef sync.(*Cond).Wait+0x7f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/cond.go:56 # 0xce377f gopkg.in/couchbase/gocbcore%2ev7.(*memdOpQueue).pop+0x3f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:138 # 0xce3032 gopkg.in/couchbase/gocbcore%2ev7.(*memdOpConsumer).Pop+0x32 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdopqueue.go:28 # 0xce4f5d gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).ioLoop+0x1cd /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:129 # 0xce5cef gopkg.in/couchbase/gocbcore%2ev7.(*memdPipelineClient).Run+0x31f /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdpipelineclient.go:195 8 @ 0x90f27a 0x91ef90 0xcf61c2 0x93cb11 # 0xcf61c1 gopkg.in/couchbase/gocbcore%2ev7.(*memdClient).run.func1+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/memdclient.go:257 6 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0x965296 0x965408 0xc69679 0xc76c01 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0x965295 io.ReadAtLeast+0x85 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:309 # 0x965407 io.ReadFull+0x57 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/io/io.go:327 # 0xc69678 github.com/couchbase/eventing/dcp/transport.(*MCRequest).Receive+0x98 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/mc_req.go:134 # 0xc76c00 github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).doReceive+0x220 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:1185 6 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0xc8aeae 0x93cb11 # 0xc8aead github.com/couchbase/eventing/dcp.(*DcpFeed).genServer+0xcd /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/upr.go:307 6 @ 0x90f27a 0x91ef90 0x10e50a0 0x93cb11 # 0x10e509f github.com/couchbase/eventing/consumer.(*Consumer).addToAggChan.func1+0x15f /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:639 6 @ 0x90f27a 0x91ef90 0xc6de6b 0x93cb11 # 0xc6de6a github.com/couchbase/eventing/dcp/transport/client.(*DcpFeed).genServer+0x22a /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/dcp/transport/client/dcp_feed.go:190 5 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb7f626 0xa26a2e 0xa26b7a 0xb800e5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb7f625 net/http.(*persistConn).Read+0x135 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1453 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb800e4 net/http.(*persistConn).readLoop+0x184 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/transport.go:1601 5 @ 0x90f27a 0x91ef90 0x1067343 0x93cb11 # 0x1067342 github.com/couchbase/eventing/suptree.(*Supervisor).Serve+0x2f2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:293 4 @ 0x90f27a 0x91ef90 0xcca98f 0x93cb11 # 0xcca98e gopkg.in/couchbase/gocbcore%2ev7.(*Agent).cccpLooper+0x55e /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocbcore.v7/agentcccpcfg.go:17 4 @ 0x90f27a 0x91ef90 0xd302f4 0x93cb11 # 0xd302f3 gopkg.in/couchbase/gocb%2ev1.(*ThresholdLoggingTracer).loggerRoutine+0x133 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/gopkg.in/couchbase/gocb.v1/thresholdlogtracer.go:244 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e04e2 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e04e1 github.com/couchbase/eventing/consumer.(*client).Serve.func2+0xd1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:115 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094875 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094874 github.com/couchbase/eventing/consumer.(*Consumer).feedbackReadMessageLoop+0x2b4 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:728 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa26fac 0x1094f9a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa26fab bufio.(*Reader).Read+0x12b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:202 # 0x1094f99 github.com/couchbase/eventing/consumer.(*Consumer).readMessageLoop+0x2a9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:792 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x11142b1 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x11142b0 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:599 3 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa90012 0xa8dfd9 0x1114c31 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa90011 net.(*UnixListener).accept+0x31 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock_posix.go:162 # 0xa8dfd8 net.(*UnixListener).Accept+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/unixsock.go:253 # 0x1114c30 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4.1+0x30 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:633 3 @ 0x90f27a 0x90f32e 0x91fc14 0x91f839 0x945fa2 0x10bebaa 0x1069903 0x93cb11 # 0x91f838 sync.runtime_Semacquire+0x38 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sema.go:56 # 0x945fa1 sync.(*WaitGroup).Wait+0x71 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/sync/waitgroup.go:129 # 0x10beba9 github.com/couchbase/eventing/consumer.(*Consumer).Serve+0x1c19 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/v8_consumer.go:292 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 3 @ 0x90f27a 0x91ef90 0x1077b8d 0x93cb11 # 0x1077b8c github.com/couchbase/eventing/consumer.(*Consumer).doLastSeqNoCheckpoint+0x19ac /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/checkpoint.go:24 3 @ 0x90f27a 0x91ef90 0x107bcec 0x93cb11 # 0x107bceb github.com/couchbase/eventing/consumer.(*Consumer).controlRoutine+0x1db /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/control_routine.go:21 3 @ 0x90f27a 0x91ef90 0x1092ade 0x93cb11 # 0x1092add github.com/couchbase/eventing/consumer.(*Consumer).sendMessageLoop+0x18d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/handle_messages.go:561 3 @ 0x90f27a 0x91ef90 0x1098377 0x93cb11 # 0x1098376 github.com/couchbase/eventing/consumer.(*Consumer).processDCPEvents+0x736 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:57 3 @ 0x90f27a 0x91ef90 0x109d9f8 0x93cb11 # 0x109d9f7 github.com/couchbase/eventing/consumer.(*Consumer).processStatsEvents+0x137 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:345 3 @ 0x90f27a 0x91ef90 0x109e883 0x93cb11 # 0x109e882 github.com/couchbase/eventing/consumer.(*Consumer).processFilterEvents+0x2e2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:395 3 @ 0x90f27a 0x91ef90 0x10a91fa 0x93cb11 # 0x10a91f9 github.com/couchbase/eventing/consumer.(*Consumer).handleFailoverLog+0x149 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:972 3 @ 0x90f27a 0x91ef90 0x10ad2be 0x93cb11 # 0x10ad2bd github.com/couchbase/eventing/consumer.(*Consumer).processReqStreamMessages+0x12d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/process_events.go:1197 3 @ 0x90f27a 0x91ef90 0x10b8f75 0x93cb11 # 0x10b8f74 github.com/couchbase/eventing/consumer.(*Consumer).updateWorkerStats+0x114 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/stats_updater.go:102 3 @ 0x90f27a 0x91ef90 0x1114485 0x93cb11 # 0x1114484 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func3+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:603 3 @ 0x90f27a 0x91ef90 0x1114e05 0x93cb11 # 0x1114e04 github.com/couchbase/eventing/producer.(*Producer).handleV8Consumer.func4+0x164 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:637 3 @ 0x950c45 0x9b1358 0x9aad3c 0x9aa2eb 0xfb298c 0x107a964 0x1069903 0x93cb11 # 0x950c44 syscall.Syscall6+0x4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/syscall/asm_linux_amd64.s:44 # 0x9b1357 os.(*Process).blockUntilWaitable+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/wait_waitid.go:31 # 0x9aad3b os.(*Process).wait+0x3b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec_unix.go:22 # 0x9aa2ea os.(*Process).Wait+0x2a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec.go:123 # 0xfb298b os/exec.(*Cmd).Wait+0x5b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/exec/exec.go:461 # 0x107a963 github.com/couchbase/eventing/consumer.(*client).Serve+0x1113 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:125 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 2 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0x9ae70e 0x9ab4aa 0xa26a2e 0xa2770c 0xa27924 0x10e023a 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0x9ae70d os.(*File).read+0x4d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file_unix.go:226 # 0x9ab4a9 os.(*File).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/file.go:107 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa2770b bufio.(*Reader).ReadSlice+0x2b /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:341 # 0xa27923 bufio.(*Reader).ReadLine+0x33 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:370 # 0x10e0239 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1c9 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:102 2 @ 0x90f27a 0x91ef90 0x104b591 0x93cb11 # 0x104b590 github.com/couchbase/eventing/util.(*Stats).updater+0x100 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/stats.go:54 2 @ 0x90f27a 0x91ef90 0x11168fc 0x93cb11 # 0x11168fb github.com/couchbase/cbauth/service.withTimeout.func1+0xdb /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 1 @ 0x8f1e12 0x923d26 0xd4d312 0x93cb11 # 0x923d25 os/signal.signal_recv+0xa5 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/sigqueue.go:139 # 0xd4d311 os/signal.loop+0x21 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/os/signal/signal_unix.go:22 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0x111678d 0x114c36e 0x11687c0 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0x111678c github.com/couchbase/cbauth/service.RegisterManager+0xbc /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:146 # 0x114c36d github.com/couchbase/eventing/service_manager.(*ServiceMgr).registerWithServer+0x11d /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:445 # 0x11687bf github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func1+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:84 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xa270b8 0xbfb0ed 0x9e08c2 0x9e05cd 0x9e0018 0xbf9c6f 0xbf7937 0xbf722a 0xbf6897 0xbfb659 0xbfbebc 0xc3f340 0xc409f5 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xa270b7 bufio.(*Reader).Read+0x237 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:216 # 0xbfb0ec github.com/couchbase/cbauth/revrpc.(*minirwc).Read+0x4c /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 # 0x9e08c1 encoding/json.(*Decoder).refill+0x131 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:159 # 0x9e05cc encoding/json.(*Decoder).readValue+0x23c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:134 # 0x9e0017 encoding/json.(*Decoder).Decode+0x77 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/encoding/json/stream.go:63 # 0xbf9c6e net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader+0x7e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/jsonrpc/server.go:66 # 0xbf7936 net/rpc.(*Server).readRequestHeader+0x66 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:589 # 0xbf7229 net/rpc.(*Server).readRequest+0x59 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:549 # 0xbf6896 net/rpc.(*Server).ServeCodec+0x96 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:464 # 0xbfb658 github.com/couchbase/cbauth/revrpc.(*Service).Run+0x538 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 # 0xbfbebb github.com/couchbase/cbauth/revrpc.BabysitService+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 # 0xc3f33f github.com/couchbase/cbauth.runRPCForSvc+0xbf /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:57 # 0xc409f4 github.com/couchbase/cbauth.startDefault.func1+0x34 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/default.go:66 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a5aad 0xa6e85f 0xa7fb3a 0xb67cf8 0xa26a2e 0xa26b7a 0xb6cf83 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a5aac internal/poll.(*FD).Read+0x17c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:157 # 0xa6e85e net.(*netFD).Read+0x4e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:202 # 0xa7fb39 net.(*conn).Read+0x69 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/net.go:176 # 0xb67cf7 net/http.(*connReader).Read+0xf7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:764 # 0xa26a2d bufio.(*Reader).fill+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:100 # 0xa26b79 bufio.(*Reader).Peek+0x39 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/bufio/bufio.go:132 # 0xb6cf82 net/http.(*conn).serve+0x7c2 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1855 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xaecf77 0xb70ed5 0xb714eb 0xb71dde 0x1168f26 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xaecf76 crypto/tls.(*listener).Accept+0x36 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/crypto/tls/tls.go:52 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb714ea net/http.(*Server).ServeTLS+0x27a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2839 # 0xb71ddd net/http.(*Server).ListenAndServeTLS+0x11d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3032 # 0x1168f25 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func4+0x3f5 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:228 1 @ 0x90f27a 0x90a59a 0x909c17 0x9a4bcb 0x9a4c4d 0x9a7048 0xa6f172 0xa894be 0xa87a99 0xb7213f 0xb70ed5 0xb70c29 0x11689a7 0x93cb11 # 0x909c16 internal/poll.runtime_pollWait+0x56 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/netpoll.go:173 # 0x9a4bca internal/poll.(*pollDesc).wait+0x9a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:85 # 0x9a4c4c internal/poll.(*pollDesc).waitRead+0x3c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_poll_runtime.go:90 # 0x9a7047 internal/poll.(*FD).Accept+0x1a7 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/internal/poll/fd_unix.go:372 # 0xa6f171 net.(*netFD).accept+0x41 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/fd_unix.go:238 # 0xa894bd net.(*TCPListener).accept+0x2d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock_posix.go:136 # 0xa87a98 net.(*TCPListener).AcceptTCP+0x48 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/tcpsock.go:246 # 0xb7213e net/http.tcpKeepAliveListener.Accept+0x2e /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:3216 # 0xb70ed4 net/http.(*Server).Serve+0x1a4 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2770 # 0xb70c28 net/http.(*Server).ListenAndServe+0xa8 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2711 # 0x11689a6 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func2+0x1b6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:186 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x117ce25 0x118820c 0x90ee22 0x93cb11 # 0x117ce24 github.com/couchbase/eventing/supervisor.(*SuperSupervisor).HandleSupCmdMsg+0x74 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:652 # 0x118820b main.main+0x3fb goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:108 # 0x90ee21 runtime.main+0x211 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/proc.go:198 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69ab 0x1183093 0x93cb11 # 0x1183092 github.com/couchbase/eventing/supervisor.NewSuperSupervisor.func1+0x72 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/supervisor/super_supervisor.go:79 1 @ 0x90f27a 0x90f32e 0x8e6cf2 0x8e69eb 0x10434c7 0x93cb11 # 0x10434c6 github.com/couchbase/eventing/util.(*vbSeqnosReader).Routine+0xf6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:100 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x1044c23 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x1044c22 github.com/couchbase/eventing/util.pollForDeletedBuckets+0x82 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/util/dcp_seqno.go:325 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x10fba14 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x10fba13 github.com/couchbase/eventing/producer.(*appLogCloser).cleanupTask+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/log_rotate.go:142 1 @ 0x90f27a 0x90f32e 0x92b7d6 0x11066dc 0x93cb11 # 0x92b7d5 time.Sleep+0x165 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/time.go:102 # 0x11066db github.com/couchbase/eventing/producer.(*Producer).updateStats+0x14b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:873 1 @ 0x90f27a 0x91ef90 0x10fe7e1 0x1069903 0x93cb11 # 0x10fe7e0 github.com/couchbase/eventing/producer.(*Producer).Serve+0x17a0 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:198 # 0x1069902 github.com/couchbase/eventing/suptree.(*Supervisor).runService.func1+0x62 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/suptree/supervisor.go:413 1 @ 0x90f27a 0x91ef90 0x1106df5 0x93cb11 # 0x1106df4 github.com/couchbase/eventing/producer.(*Producer).pollForDeletedVbs+0x134 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/producer/producer.go:897 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1151cf2 0x1116bc4 0x1115feb 0x11163e3 0x1117582 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1151cf1 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetTaskList+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:37 # 0x1116bc3 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:99 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x11163e2 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:98 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0x114cf67 0x1152482 0x1116a14 0x1115feb 0x1116283 0x11174b2 0x93a3cb 0x98c499 0x98ba14 0xbf5fbe 0x93cb11 # 0x114cf66 github.com/couchbase/eventing/service_manager.(*ServiceMgr).wait+0x2a6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:542 # 0x1152481 github.com/couchbase/eventing/service_manager.(*ServiceMgr).GetCurrentTopology+0x1a1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/service_manager.go:90 # 0x1116a13 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1+0x73 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:76 # 0x1115fea github.com/couchbase/cbauth/service.withTimeout+0x7a /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:46 # 0x1116282 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology+0xd2 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/service/revrpc.go:75 # 0x93a3ca runtime.call64+0x3a /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/asm_amd64.s:574 # 0x98c498 reflect.Value.call+0x968 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:447 # 0x98ba13 reflect.Value.Call+0xa3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/reflect/value.go:308 # 0xbf5fbd net/rpc.(*service).call+0x14d /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/rpc/server.go:384 1 @ 0x90f27a 0x91ef90 0xb98e91 0x93cb11 # 0xb98e90 github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop+0xe0 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 1 @ 0x90f27a 0x91ef90 0xb992ff 0x93cb11 # 0xb992fe github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop+0xce /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116924a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169249 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func5+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:242 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x11693ca 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x11693c9 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func6+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:253 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x116954a 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1169549 github.com/couchbase/eventing/service_manager.(*ServiceMgr).initService.func7+0x99 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:264 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188939 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188938 main.main.func1+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:40 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188a49 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188a48 main.main.func2+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:52 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188b59 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188b58 main.main.func3+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:64 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188c69 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188c68 main.main.func4+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:76 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188d79 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188d78 main.main.func5+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:88 1 @ 0x90f27a 0x91ef90 0xc4333a 0xc42ff8 0xc43ab5 0x1188e89 0x93cb11 # 0xc43339 github.com/couchbase/cbauth/metakv.doRunObserveChildren+0x2f9 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:287 # 0xc42ff7 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren+0x57 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:246 # 0xc43ab4 github.com/couchbase/cbauth/metakv.RunObserveChildren+0x54 /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/cbauth/metakv/metakv.go:370 # 0x1188e88 main.main.func6+0x88 goproj/src/github.com/couchbase/eventing/cmd/producer/main.go:100 1 @ 0x952e8b 0x955473 0x953677 0xc46687 0xc46c5c 0x10e0229 0x93cb11 # 0x952e8a time.appendInt+0x2ea /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/time/format.go:354 # 0x955472 time.Time.AppendFormat+0x1d62 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/time/format.go:582 # 0x953676 time.Time.Format+0xb6 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/time/format.go:493 # 0xc46686 github.com/couchbase/eventing/logging.printf+0xa6 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/logging/logging.go:109 # 0xc46c5b github.com/couchbase/eventing/logging.Infof+0x5b /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/logging/logging.go:141 # 0x10e0228 github.com/couchbase/eventing/consumer.(*client).Serve.func1+0x1b8 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/consumer/client.go:108 1 @ 0xb677f0 0x93cb11 # 0xb677f0 net/http.(*connReader).backgroundRead+0x0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:667 1 @ 0xd44938 0xd44740 0xd41284 0xd4ca4d 0xd4cdd1 0x11463ad 0x1169fb8 0xb6ddf4 0xb6fa60 0xb70b0c 0xb6ce11 0x93cb11 # 0xd44937 runtime/pprof.writeRuntimeProfile+0x97 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:679 # 0xd4473f runtime/pprof.writeGoroutine+0x9f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:641 # 0xd41283 runtime/pprof.(*Profile).WriteTo+0x3e3 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/runtime/pprof/pprof.go:310 # 0xd4ca4c net/http/pprof.handler.ServeHTTP+0x20c /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:243 # 0xd4cdd0 net/http/pprof.Index+0x1d0 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/pprof/pprof.go:254 # 0x11463ac github.com/couchbase/eventing/service_manager.(*ServiceMgr).indexHandler+0x8c /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/http_handlers.go:3578 # 0x1169fb7 github.com/couchbase/eventing/service_manager.(*ServiceMgr).(github.com/couchbase/eventing/service_manager.indexHandler)-fm+0x47 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/service_manager/manager.go:99 # 0xb6ddf3 net/http.HandlerFunc.ServeHTTP+0x43 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1947 # 0xb6fa5f net/http.(*ServeMux).ServeHTTP+0x12f /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2337 # 0xb70b0b net/http.serverHandler.ServeHTTP+0xbb /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:2694 # 0xb6ce10 net/http.(*conn).serve+0x650 /home/couchbase/.cbdepscache/exploded/x86_64/go-1.10.3/go/src/net/http/server.go:1830 =================================================================================================================================================== [2019-12-02 07:52:59,695] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.242 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root, ip:172.23.105.238 port:8091 ssh_username:root] [2019-12-02 07:52:59,722] - [eventing_base] [139937039668992] - INFO - Stats for Node 172.23.105.242 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 14, "agg_messages_sent_to_worker": 31561, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 6277, "dcp_deletion_sent_to_worker": 6277, "dcp_mutation": 3341, "dcp_mutation_sent_to_worker": 3341, "dcp_snapshot": 7008, "dcp_stream_req_counter": 503, "dcp_stream_req_err_counter": 2, "dcp_streamend": 160, "dcp_streamreq": 495, "execution_stats": 4254, "failure_stats": 4254, "latency_stats": 4254, "lcb_exception_stats": 4254, "log_level": 3, "num_processed_events": 9618, "processed_events_size": 3914220, "thr_count": 3, "thr_map": 3, "timer_events": 12349, "timer_responses_received": 9446, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 6277, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3341, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 6277, "enqueued_dcp_mutation_msg_counter": 3341, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 172, "lcb_retry_failure": 0, "messages_parsed": 31549, "num_processed_events": 9618, "on_delete_failure": 0, "on_delete_success": 6277, "on_update_failure": 0, "on_update_success": 3169, "processed_events_size": 3914220, "timer_create_counter": 9446, "timer_create_failure": 0, "timer_msg_counter": 12349, "timer_responses_sent": 0, "timestamp": { "62135": "2019-12-02T15:52:59Z", "62141": "2019-12-02T15:52:59Z", "62142": "2019-12-02T15:52:59Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 356, "timeout_count": 12, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "62135": "2019-12-02T15:52:59Z", "62141": "2019-12-02T15:52:59Z", "62142": "2019-12-02T15:52:59Z" }, "v8worker_events_lost": 0 }, "function_id": 1571568087, "function_name": "Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance", "gocb_creds_request_counter": 20, "internal_vb_distribution_stats": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "latency_percentile_stats": { "100": 5355600, "50": 1400, "80": 2100, "90": 2800, "95": 3700, "99": 6900 }, "lcb_creds_request_counter": 132, "lcb_exception_stats": { "12009": 356 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.238:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.105.242:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.105.24:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.105.238:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "172.23.105.242:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "172.23.105.24:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" } }, "worker_pids": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": 62135, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": 62141, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": 62142 } } ] [2019-12-02 07:52:59,746] - [eventing_base] [139937039668992] - INFO - Stats for Node 172.23.105.24 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 38101, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 6257, "dcp_deletion_sent_to_worker": 6257, "dcp_mutation": 7828, "dcp_mutation_sent_to_worker": 7828, "dcp_snapshot": 12585, "dcp_stream_req_counter": 1197, "dcp_stream_req_err_counter": 3, "dcp_streamend": 853, "dcp_streamreq": 1194, "execution_stats": 4389, "failure_stats": 4389, "latency_stats": 4389, "lcb_exception_stats": 4389, "log_level": 3, "num_processed_events": 14085, "processed_events_size": 6927828, "thr_count": 3, "thr_map": 3, "timer_events": 12534, "timer_responses_received": 14084, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 6257, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 7828, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 6257, "enqueued_dcp_mutation_msg_counter": 7828, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 1, "lcb_retry_failure": 0, "messages_parsed": 38089, "num_processed_events": 14085, "on_delete_failure": 0, "on_delete_success": 6257, "on_update_failure": 0, "on_update_success": 7827, "processed_events_size": 6927828, "timer_create_counter": 14084, "timer_create_failure": 0, "timer_msg_counter": 12534, "timer_responses_sent": 0, "timestamp": { "59939": "2019-12-02T15:52:59Z", "59944": "2019-12-02T15:52:59Z", "59946": "2019-12-02T15:52:59Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 81100, "timeout_count": 45, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "59939": "2019-12-02T15:52:59Z", "59944": "2019-12-02T15:52:59Z", "59946": "2019-12-02T15:52:59Z" }, "v8worker_events_lost": 0 }, "function_id": 1571568087, "function_name": "Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance", "gocb_creds_request_counter": 20, "internal_vb_distribution_stats": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" }, "latency_percentile_stats": { "100": 2501600, "50": 1400, "80": 2200, "90": 2900, "95": 3700, "99": 6400 }, "lcb_creds_request_counter": 152, "lcb_exception_stats": { "12009": 75680, "47": 5420 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.238:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.105.242:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.105.24:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.105.238:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "172.23.105.242:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "172.23.105.24:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" } }, "worker_pids": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": 59939, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": 59944, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": 59946 } } ] [2019-12-02 07:52:59,774] - [eventing_base] [139937039668992] - INFO - Stats for Node 172.23.105.238 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 10, "agg_messages_sent_to_worker": 23268, "agg_queue_memory": 996, "agg_queue_memory_cap": 106954752, "agg_queue_size": 249, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 6263, "dcp_deletion_sent_to_worker": 6263, "dcp_mutation": 3172, "dcp_mutation_sent_to_worker": 3172, "dcp_snapshot": 6601, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 2694, "failure_stats": 2694, "latency_stats": 2694, "lcb_exception_stats": 2694, "log_level": 3, "num_processed_events": 9435, "processed_events_size": 3792040, "thr_count": 3, "thr_map": 3, "timer_events": 87, "timer_responses_received": 9435, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 996, "agg_queue_size": 249, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 6263, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3172, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 6263, "enqueued_dcp_mutation_msg_counter": 3172, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 23256, "num_processed_events": 9435, "on_delete_failure": 0, "on_delete_success": 6263, "on_update_failure": 0, "on_update_success": 3172, "processed_events_size": 3792040, "timer_create_counter": 9435, "timer_create_failure": 0, "timer_msg_counter": 87, "timer_responses_sent": 0, "timestamp": { "130991": "2019-12-02T15:52:59Z", "130992": "2019-12-02T15:52:59Z", "131002": "2019-12-02T15:52:59Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 320015, "timeout_count": 84, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "130991": "2019-12-02T15:52:59Z", "130992": "2019-12-02T15:52:59Z", "131002": "2019-12-02T15:52:59Z" }, "v8worker_events_lost": 0 }, "function_id": 1571568087, "function_name": "Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance", "gocb_creds_request_counter": 12, "internal_vb_distribution_stats": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "latency_percentile_stats": { "100": 21400, "50": 1300, "80": 2000, "90": 2700, "95": 3500, "99": 6100 }, "lcb_creds_request_counter": 108, "lcb_exception_stats": { "12009": 255969, "47": 64047 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.238:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.105.242:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.105.24:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.105.238:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[0-113]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[114-227]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[228-341]" }, "172.23.105.242:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[342-455]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[456-569]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[570-682]" }, "172.23.105.24:8096": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": "[683-796]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": "[797-910]", "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": "[911-1023]" } }, "worker_pids": { "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_0": 130991, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_1": 130992, "worker_Function_856920210_test_erl_crash_on_kv_and_eventing_node_during_eventing_rebalance_2": 131002 } } ] [2019-12-02 07:52:59,806] - [basetestcase] [139937039668992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.242 port:8091 ssh_username:root, ip:172.23.105.24 port:8091 ssh_username:root, ip:172.23.105.238 port:8091 ssh_username:root] [2019-12-02 07:52:59,810] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:53:05,406] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:53:06,188] - [rest_client] [139937039668992] - INFO - /diag/eval status on 172.23.105.242:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-12-02 07:53:06,189] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2019-12-02 07:53:06,471] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:53:06,471] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2019-12-02 07:53:06,521] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:53:06,639] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:53:12,246] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:53:13,064] - [rest_client] [139937039668992] - INFO - /diag/eval status on 172.23.105.24:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-12-02 07:53:13,065] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2019-12-02 07:53:14,565] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:53:14,565] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2019-12-02 07:53:14,584] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:53:14,700] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:53:20,492] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:53:21,255] - [rest_client] [139937039668992] - INFO - /diag/eval status on 172.23.105.238:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2019-12-02 07:53:21,255] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2019-12-02 07:53:23,540] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:53:23,540] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2019-12-02 07:53:23,561] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:53:23,774] - [eventing_base] [139937039668992] - INFO - Bucket dst_bucket DGM is 100 [2019-12-02 07:53:23,817] - [eventing_base] [139937039668992] - INFO - Bucket metadata DGM is 100 [2019-12-02 07:53:23,856] - [eventing_base] [139937039668992] - INFO - Bucket src_bucket DGM is 100 [2019-12-02 07:53:23,878] - [basetestcase] [139937039668992] - INFO - ------- Cluster statistics ------- [2019-12-02 07:53:23,878] - [basetestcase] [139937039668992] - INFO - 172.23.105.236:8091 => {'swap_mem_used': 1048576, 'cpu_utilization': 7.692307692307693, 'mem_free': 2879987712, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'kv']} [2019-12-02 07:53:23,878] - [basetestcase] [139937039668992] - INFO - 172.23.105.240:8091 => {'swap_mem_used': 3932160, 'cpu_utilization': 35.67567567567568, 'mem_free': 3261313024, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'index', u'n1ql']} [2019-12-02 07:53:23,878] - [basetestcase] [139937039668992] - INFO - 172.23.105.242:8091 => {'swap_mem_used': 1310720, 'cpu_utilization': 3.526448362720403, 'mem_free': 3342995456, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'eventing']} [2019-12-02 07:53:23,878] - [basetestcase] [139937039668992] - INFO - 172.23.105.24:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.262626262626263, 'mem_free': 3301740544, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'eventing']} [2019-12-02 07:53:23,879] - [basetestcase] [139937039668992] - INFO - 172.23.105.237:8091 => {'swap_mem_used': 0, 'cpu_utilization': 7.614213197969543, 'mem_free': 2836660224, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'kv']} [2019-12-02 07:53:23,879] - [basetestcase] [139937039668992] - INFO - 172.23.105.238:8091 => {'swap_mem_used': 2359296, 'cpu_utilization': 72.09302325581395, 'mem_free': 3322134528, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'eventing']} [2019-12-02 07:53:23,879] - [basetestcase] [139937039668992] - INFO - --- End of cluster statistics --- [2019-12-02 07:53:23,882] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:53:29,158] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:53:30,133] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:53:35,538] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:53:36,524] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:53:41,978] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:53:43,134] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:53:49,485] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:53:50,470] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:53:56,636] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:53:57,601] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:54:03,030] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:54:03,960] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:54:09,690] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:54:10,730] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:54:16,658] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:54:17,583] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:54:23,233] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:54:23,946] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:54:29,390] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:54:30,236] - [remote_util] [139937039668992] - INFO - 172.23.105.236 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:54:30,427] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:54:30,428] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: systemctl stop couchbase-server.service [2019-12-02 07:54:37,612] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:54:37,612] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:54:37,726] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:54:38,496] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:54:38,496] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:54:38,521] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:54:38,525] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:54:43,975] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.236 [2019-12-02 07:54:44,789] - [remote_util] [139937039668992] - INFO - 172.23.105.236 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:54:44,913] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:54:44,914] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.236: systemctl start couchbase-server.service [2019-12-02 07:54:44,949] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:54:44,949] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:54:45,066] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:54:50,485] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:54:51,263] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:54:57,051] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:54:57,943] - [remote_util] [139937039668992] - INFO - 172.23.105.237 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:54:58,071] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:54:58,072] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: systemctl stop couchbase-server.service [2019-12-02 07:55:00,946] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:00,946] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:55:01,060] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:55:01,880] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:01,880] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:55:01,902] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:01,905] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:55:07,641] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.237 [2019-12-02 07:55:08,433] - [remote_util] [139937039668992] - INFO - 172.23.105.237 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:55:08,561] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:55:08,562] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.237: systemctl start couchbase-server.service [2019-12-02 07:55:08,597] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:08,597] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:55:08,715] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:55:14,076] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:55:14,837] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:55:20,877] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:55:21,711] - [remote_util] [139937039668992] - INFO - 172.23.105.238 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:55:21,841] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:55:21,841] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: systemctl stop couchbase-server.service [2019-12-02 07:55:23,562] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:23,563] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:55:23,677] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:55:23,795] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:23,795] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:55:23,814] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:23,818] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 07:55:29,101] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.238 [2019-12-02 07:55:29,936] - [remote_util] [139937039668992] - INFO - 172.23.105.238 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:55:30,064] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:55:30,065] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.238: systemctl start couchbase-server.service [2019-12-02 07:55:30,100] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:30,101] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:55:30,218] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:55:35,774] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:55:36,692] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:55:44,649] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:55:45,466] - [remote_util] [139937039668992] - INFO - 172.23.105.24 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:55:45,593] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:55:45,593] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: systemctl stop couchbase-server.service [2019-12-02 07:55:47,314] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:47,315] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:55:47,429] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:55:47,464] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:47,464] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:55:47,546] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:47,549] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 07:55:53,350] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.24 [2019-12-02 07:55:54,159] - [remote_util] [139937039668992] - INFO - 172.23.105.24 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:55:54,343] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:55:54,344] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.24: systemctl start couchbase-server.service [2019-12-02 07:55:54,379] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:55:54,379] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:55:54,496] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:56:00,427] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:56:01,190] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:56:06,566] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:56:07,389] - [remote_util] [139937039668992] - INFO - 172.23.105.240 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:56:07,518] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:56:07,518] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: systemctl stop couchbase-server.service [2019-12-02 07:56:09,490] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:09,490] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:56:09,555] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:56:09,574] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:09,575] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:56:09,656] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:09,660] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 07:56:15,011] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.240 [2019-12-02 07:56:15,853] - [remote_util] [139937039668992] - INFO - 172.23.105.240 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:56:15,978] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:56:15,979] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.240: systemctl start couchbase-server.service [2019-12-02 07:56:16,014] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:16,014] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:56:16,132] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:56:21,504] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:56:22,306] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:56:27,824] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:56:28,616] - [remote_util] [139937039668992] - INFO - 172.23.105.242 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:56:28,744] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:56:28,744] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: systemctl stop couchbase-server.service [2019-12-02 07:56:30,466] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:30,466] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:56:30,580] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:56:30,600] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:30,600] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:56:30,681] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:30,685] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 07:56:36,027] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.242 [2019-12-02 07:56:36,829] - [remote_util] [139937039668992] - INFO - 172.23.105.242 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:56:36,954] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:56:36,955] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.242: systemctl start couchbase-server.service [2019-12-02 07:56:36,990] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:36,990] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:56:37,108] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:56:42,952] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:56:43,861] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:56:49,846] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:56:50,634] - [remote_util] [139937039668992] - INFO - 172.23.105.244 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:56:50,761] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:56:50,761] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: systemctl stop couchbase-server.service [2019-12-02 07:56:52,432] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:52,433] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:56:52,547] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:56:52,566] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:52,567] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:56:52,648] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:52,652] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 07:56:58,300] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.244 [2019-12-02 07:56:59,158] - [remote_util] [139937039668992] - INFO - 172.23.105.244 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:56:59,216] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:56:59,216] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.244: systemctl start couchbase-server.service [2019-12-02 07:56:59,251] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:56:59,251] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:56:59,367] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:57:05,102] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:57:05,902] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:57:11,071] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:57:11,861] - [remote_util] [139937039668992] - INFO - 172.23.105.245 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:57:11,980] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:57:11,980] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: systemctl stop couchbase-server.service [2019-12-02 07:57:13,650] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:57:13,650] - [basetestcase] [139937039668992] - INFO - Couchbase stopped [2019-12-02 07:57:13,764] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2019-12-02 07:57:13,782] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:57:13,782] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2019-12-02 07:57:13,863] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:57:13,865] - [remote_util] [139937039668992] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 07:57:19,148] - [remote_util] [139937039668992] - INFO - Connected to 172.23.105.245 [2019-12-02 07:57:19,930] - [remote_util] [139937039668992] - INFO - 172.23.105.245 **** The version file /opt/couchbase/ VERSION.txt exists [2019-12-02 07:57:20,054] - [remote_util] [139937039668992] - INFO - Running systemd command on this server [2019-12-02 07:57:20,055] - [remote_util] [139937039668992] - INFO - running command.raw on 172.23.105.245: systemctl start couchbase-server.service [2019-12-02 07:57:20,090] - [remote_util] [139937039668992] - INFO - command executed successfully [2019-12-02 07:57:20,091] - [basetestcase] [139937039668992] - INFO - Couchbase started [2019-12-02 07:57:20,205] - [basetestcase] [139937039668992] - INFO - sleep for 10 secs. ... [2019-12-02 07:57:30,468] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.236 with username:root [2019-12-02 07:57:35,908] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.236 [2019-12-02 07:57:36,641] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.236: /opt/couchbase/bin/cbcollect_info 172.23.105.236-20191202-0757-diag.zip [2019-12-02 07:59:17,412] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 07:59:21,472] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.236: rm -f /root/172.23.105.236-20191202-0757-diag.zip [2019-12-02 07:59:21,570] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 07:59:21,688] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.237 with username:root [2019-12-02 07:59:28,009] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.237 [2019-12-02 07:59:28,748] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.237: /opt/couchbase/bin/cbcollect_info 172.23.105.237-20191202-0759-diag.zip [2019-12-02 08:01:03,337] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:01:05,839] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.237: rm -f /root/172.23.105.237-20191202-0759-diag.zip [2019-12-02 08:01:05,939] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:01:06,057] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.238 with username:root [2019-12-02 08:01:12,031] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.238 [2019-12-02 08:01:12,791] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.238: /opt/couchbase/bin/cbcollect_info 172.23.105.238-20191202-0801-diag.zip [2019-12-02 08:02:45,175] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:02:47,505] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.238: rm -f /root/172.23.105.238-20191202-0801-diag.zip [2019-12-02 08:02:47,603] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:02:47,722] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.24 with username:root [2019-12-02 08:02:54,143] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.24 [2019-12-02 08:02:54,897] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.24: /opt/couchbase/bin/cbcollect_info 172.23.105.24-20191202-0802-diag.zip [2019-12-02 08:04:25,694] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:04:27,512] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.24: rm -f /root/172.23.105.24-20191202-0802-diag.zip [2019-12-02 08:04:27,610] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:04:27,728] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.240 with username:root [2019-12-02 08:04:33,795] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.240 [2019-12-02 08:04:34,516] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.240: /opt/couchbase/bin/cbcollect_info 172.23.105.240-20191202-0804-diag.zip [2019-12-02 08:06:08,511] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:06:10,505] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.240: rm -f /root/172.23.105.240-20191202-0804-diag.zip [2019-12-02 08:06:10,603] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:06:10,721] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.242 with username:root [2019-12-02 08:06:16,438] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.242 [2019-12-02 08:06:17,198] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.242: /opt/couchbase/bin/cbcollect_info 172.23.105.242-20191202-0806-diag.zip [2019-12-02 08:07:47,514] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:07:49,580] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.242: rm -f /root/172.23.105.242-20191202-0806-diag.zip [2019-12-02 08:07:49,679] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:07:49,798] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.244 with username:root [2019-12-02 08:07:55,696] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.244 [2019-12-02 08:07:56,449] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.244: /opt/couchbase/bin/cbcollect_info 172.23.105.244-20191202-0807-diag.zip [2019-12-02 08:09:29,889] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:09:31,777] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.244: rm -f /root/172.23.105.244-20191202-0807-diag.zip [2019-12-02 08:09:31,875] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:09:31,994] - [remote_util] [139937318188864] - INFO - connecting to 172.23.105.245 with username:root [2019-12-02 08:09:38,070] - [remote_util] [139937318188864] - INFO - Connected to 172.23.105.245 [2019-12-02 08:09:38,762] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.245: /opt/couchbase/bin/cbcollect_info 172.23.105.245-20191202-0809-diag.zip [2019-12-02 08:11:21,956] - [remote_util] [139937318188864] - INFO - command executed successfully [2019-12-02 08:11:22,374] - [remote_util] [139937318188864] - INFO - running command.raw on 172.23.105.245: rm -f /root/172.23.105.245-20191202-0809-diag.zip [2019-12-02 08:11:22,395] - [remote_util] [139937318188864] - INFO - command executed successfully