Details
Description
Build - 7.0.2 - 6512
STEPS
- Cluster has 1 kv and 1 eventing node.
- disable auto failover and enable n2n encryption.
- Create and deploy eventing handler.
- Load docs into source bucket, mutations are processed.
- Perform swap rebalance of eventing nodes.
2021-08-16T12:18:49.606Z, ns_orchestrator:0:info:message(ns_1@10.112.190.101) - Starting rebalance, KeepNodes = ['ns_1@10.112.190.101','ns_1@10.112.190.102'], EjectNodes = ['ns_1@10.112.190.103'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 0ee8a6a92a230c47127f4a9d0ef4c256
- While rebalance is ongoing enforce tls.
2021-08-16T12:18:54.079+00:00 [Info] Updating node-to-node encryption level: {EncryptData:true DisableNonSSLPorts:true}
OBSERVATION
Rebalance hung.
On 10.112.190.102
eventing.log
2021-08-16T12:18:54.079+00:00 [Error] Eventing::main metakv observe error for global config, err: unexpected EOF. Retrying... |
2021-08-16T12:18:54.079+00:00 [Warn] servicesChangeNotifier: Connection terminated for collection manifest notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default, bucket: dst_bucket, (unexpected EOF). Retrying... |
2021-08-16T12:18:54.079+00:00 [Warn] servicesChangeNotifier: Connection terminated for collection manifest notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default, bucket: metadata, (unexpected EOF). Retrying... |
2021-08-16T12:18:54.079+00:00 [Warn] servicesChangeNotifier: Connection terminated for services notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default (unexpected EOF). Retrying... |
2021-08-16T12:18:54.079+00:00 [Warn] servicesChangeNotifier: Connection terminated for collection manifest notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default, bucket: src_bucket, (unexpected EOF). Retrying... |
2021-08-16T12:18:54.079+00:00 [Warn] servicesChangeNotifier: Connection terminated for pool notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default (unexpected EOF). Retrying... |
2021-08-16T12:18:54.079+00:00 [Error] Eventing::main metakv observe error for apps retry, err: unexpected EOF. Retrying. |
2021-08-16T12:18:54.079+00:00 [Error] Eventing::main metakv observe error for settings, err: unexpected EOF. Retrying... |
2021-08-16T12:18:54.079+00:00 [Error] Eventing::main metakv observe error for event handler code, err: unexpected EOF. Retrying... |
2021-08-16T12:18:54.079+00:00 [Error] ServiceMgr::initService metakv observe error for primary store, err: unexpected EOF. Retrying... |
2021-08-16T12:18:54.079+00:00 [Error] ServiceMgr::initService metakv observe error for temp store, err: unexpected EOF. Retrying... |
2021-08-16T12:18:54.079+00:00 [Error] ServiceMgr::initService metakv observe error for setting store, err: unexpected EOF. Retrying... |
2021-08-16T12:18:54.079+0000 [WARN] Store::ObserveChanges : Unable to observe metakv unexpected EOF ..Retrying |
2021-08-16T12:18:54.080+00:00 [Error] Eventing::main metakv observe error for debugger, err: unexpected EOF. Retrying. |
2021-08-16T12:18:54.080+00:00 [Error] ServiceMgr::getConfig failed to get config, err: Get http://127.0.0.1:8091/_metakv/eventing/settings/config: dial tcp 127.0.0.1:8091: connect: connection refused |
Attachments
Issue Links
- duplicates
-
MB-47946 [Eventing][enforce-tls]: handler stuck in deploying state
-
- Closed
-