[BP 7.0.2 MB-47946] - [Eventing][enforce-tls]: handler stuck in deploying state

Build - 7.0.2 - 6504

STEPS TO REPRODUCE

• Disable auto failover and enable n2n encryption.
• Create and deploy eventing handler.
• Load docs into source bucket, mutations are processed.
• Pause eventing handler.
• Resume eventing handler.
• While handler is in deploying state, enforce tls by changing ClusterEncryptionLevel to strict.

Handler is forever stuck in deploying state.

On 10.112.190.102

• eventing.log

  2021-08-13T11:25:15.146+00:00 [Info] Cluster Encryption Settings have been changed by ns server.

  2021-08-13T11:25:15.146+00:00 [Info] Updating node-to-node encryption level: {EncryptData:true DisableNonSSLPorts:true}

  2021-08-13T11:25:15.146+00:00 [Info] Attempting to restart HTTP server to listen on loopback interface

  2021-08-13T11:25:15.146+00:00 [Info] Successfully stopped running HTTP server

  2021-08-13T11:25:15.146+00:00 [Info] ServiceMgr::initService Got a signal to stop running HTTP server

  2021-08-13T11:25:15.146+00:00 [Info] serviceChangeNotifier: received EncryptionLevelChangeNotification

  2021-08-13T11:25:15.146+00:00 [Info] ServiceMgr::initService Admin HTTP server started: 127.0.0.1:8096

  2021-08-13T11:25:15.146+00:00 [Error] ServiceMgr::initService metakv observe error for primary store, err: unexpected EOF. Retrying...

  2021-08-13T11:25:15.146+00:00 [Error] ServiceMgr::initService metakv observe error for temp store, err: unexpected EOF. Retrying...

  2021-08-13T11:25:15.146+00:00 [Error] Eventing::main metakv observe error for rebalance token, err: unexpected EOF. Retrying...

  2021-08-13T11:25:15.146+00:00 [Error] ServiceMgr::initService metakv observe error for setting store, err: unexpected EOF. Retrying...

  2021-08-13T11:25:15.146+00:00 [Error] Eventing::main metakv observe error for global config, err: unexpected EOF. Retrying...

  2021-08-13T11:25:15.146+00:00 [Error] Eventing::main metakv observe error for debugger, err: unexpected EOF. Retrying.

  2021-08-13T11:25:15.146+00:00 [Error] Eventing::main metakv observe error for apps retry, err: unexpected EOF. Retrying.

  2021-08-13T11:25:15.146+00:00 [Error] Eventing::main metakv observe error for event handler code, err: unexpected EOF. Retrying...

  2021-08-13T11:25:15.146+0000 [WARN] Store::ObserveChanges : Unable to observe metakv unexpected EOF ..Retrying

  2021-08-13T11:25:15.146+00:00 [Warn] servicesChangeNotifier: Connection terminated for pool notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default (unexpected EOF). Retrying...

  2021-08-13T11:25:15.146+00:00 [Warn] servicesChangeNotifier: Connection terminated for services notifier instance of http://%40eventing-cbauth@127.0.0.1:8091, default (unexpected EOF). Retrying...

  2021-08-13T11:25:15.262+00:00 [Error] [gocb] memdClient read failure: EOF