Critical Description
While investigating MB-62181, I came across the following cascading failure when enabling n2n encryption with the "all" setting and setting TLS certificate requirements to "mandatory"
2024-06-10T15:36:37.995+05:30 [WARN] (GOCBCORE) 0x14007141340 memdClient read failure on conn `1af0237dfea2ca93/7757d0b515457cf2` : remote error: tls: certificate required -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:742 |
2024-06-10T15:36:37.996+05:30 [WARN] (GOCBCORE) Pipeline Client 0x1400a339500 failed to bootstrap: EOF | {"document_key":"{\"a\":\"gocbcore/v10.2.10 fts:stats\",\"i\":\"1af0237dfea2ca93/7757d0b515457cf2\"}","bucket":"bkt1","last_dispatched_to":"127.0.0.1:11990","last_dispatched_from":"127.0.0.1:61334","last_connection_id":"1af0237dfea2ca93/7757d0b515457cf2"} -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:742 |
2024-06-10T15:36:37.996+05:30 [WARN] (GOCBCORE) Pipeline Client 0x1400a3394a0 failed to bootstrap: bucket not found -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:742 |
2024-06-10T15:36:38.002+05:30 [WARN] janitor: JanitorOnce, err: janitor: JanitorOnce errors: 1, []string{"#0: janitor: adding feed, err: feed_dcp_gocbcore: StartGocbcoreDCPFeed, could not prepare DCP feed, name: bkt1._default.test3_cc1c1b0282317eb5_4c1c5584, server: http://127.0.0.1:9000, bucketName: bkt1, indexName: bkt1._default.test3, err: newGocbcoreDCPFeed: error in setting up feed's stream options, err: agent setup failed, err: gocbcore_utils: createAgents (1), setup err: agent setup failed, err: EOF | {\"document_key\":\"{\\\"a\\\":\\\"gocbcore/v10.2.10 fts:stats\\\",\\\"i\\\":\\\"1af0237dfea2ca93/7757d0b515457cf2\\\"}\",\"bucket\":\"bkt1\",\"last_dispatched_to\":\"127.0.0.1:11990\",\"last_dispatched_from\":\"127.0.0.1:61334\",\"last_connection_id\":\"1af0237dfea2ca93/7757d0b515457cf2\"}"} -- cbgt.(*Manager).JanitorLoop() at manager_janitor.go:204 |
this resulted in the closing of existing DCP feeds.
Steps to reproduce:
- Create a 2 node cluster with a 2 partitioned index.
- Disable auto failover using this Couchbase CLI command -
curl -X POST http://localhost:9000/settings/autoFailover -u Administrator:asdasd -d enabled=false |
3. Enable n2n encryption with the following command -
./couchbase-cli node-to-node-encryption --cluster localhost:9000 --username Administrator --password asdasd --enable |
4. In the UI, navigate to Security > Other Settings > Cluster Encryption > "all"
5. Then, change the certificate requirement: Security > Certificates > Require Client Certificate > "Mandatory".
6. The FTS logs should show repeated errors.