Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
7.6.0, 7.2.4
-
Operating System : Debian GNU/Linux 11 (bullseye)
-
Untriaged
-
Linux x86_64
-
0
-
Unknown
Description
Case-1
In 7.2.4 Cluster
- Created a single node cluster 172.23.104.102 on Couchbase Enterprise Edition 7.2.4-7070
- Added nodes 172.23.109.199, 172.23.216.178, 172.23.109.34 and 172.23.109.65
- The /pools/default/terseClusterInfo was queried and the orchestrator was found
- The endpoint /pools/default was queried
/pools/default/terseClusterInfo response
{u'clusterUUID': u'f49dca3b6a0f8fb9ad86557d65f81844', u'clusterCompatVersion': u'7.2', u'isBalanced': False, u'orchestrator': u'ns_1@172.23.104.102'} |
/pools/default returns many fields with "unknown" for the node 172.23.109.65. Also displays status as unhealthy
{u'nodeEncryption': False, u'configuredHostname': u'172.23.109.65:8091', u'externalListeners': [{u'nodeEncryption': False, u'afamily': u'inet'}], u'memoryFree': 0, u'serverGroup': u'Group 1', u'ports': {u'direct': 11210, u'httpsMgmt': 18091, u'distTLS': 21150, u'httpsCAPI': 18092, u'distTCP': 21100}, u'nodeUUID': u'3325698e4e8f0cbe30d8f915247b73d2', u'recoveryType': u'none', u'hostname': u'172.23.109.65:8091', u'otpNode': u'ns_1@172.23.109.65', u'clusterCompatibility': 458754, u'nodeHash': 115932406, u'addressFamily': u'inet', u'mcdMemoryReserved': 0, u'cpuCount': u'unknown', u'os': u'unknown', u'couchApiBase': u'http://172.23.109.65:8092/', u'systemStats': {}, u'mcdMemoryAllocated': 0, u'services': [u'cbas'], u'version': u'unknown', u'clusterMembership': u'inactiveAdded', u'uptime': u'0', u'interestingStats': {}, u'addressFamilyOnly': False, u'couchApiBaseHTTPS': u'https://172.23.109.65:18092/', u'memoryTotal': 0, u'status': u'unhealthy'}, {u'nodeEncryption': False, u'configuredHostname': u'172.23.216.178:8091', u'externalListeners': [{u'nodeEncryption': False, u'afamily': u'inet'} |
After a couple of minutes, the pools/default response contains all expected values for the fields
/pools/default response right after add node -https://cb-engineering.s3.amazonaws.com/MB-60705/pools_default_neo_unknown.json
/pools/default response after a couple of minutes - https://cb-engineering.s3.amazonaws.com/MB-60705/pools_default_neo_expected.json
Cluster logs
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063128-ns_1%40172.23.109.199.zip
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063128-ns_1%40172.23.109.34.zip
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063128-ns_1%40172.23.109.65.zip
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063128-ns_1%40172.23.216.178.zip
Case-2
In a mixed mode Trinity cluster
- Created a 5 node cluster on Couchbase Enterprise Edition build 7.1.2-3454
- 172.23.123.235 - cbas
- 172.23.216.65 - kv, index, n1ql
- 172.23.121.169 - cbas
- 172.23.107.53 - cbas
- 172.23.123.213 - kv,index,n1ql
- Created bucket loaded some data
- Created datasets, etc and ran a few analytics queries
- Upgraded using swap rebalance rebalanced in node 172.23.121.166 and removed node 172.23.107.53 - 172.23.121.166 was with Couchbase Enterprise Edition build 7.6.0-2102
- Upgraded another node using swap rebalance - 172.23.121.169 was removed and 172.23.107.53 was added
- Post this a new node 172.23.121.169 with Couchbase Enterprise Edition build 7.6.0-2102 was added
- The /pools/default/terseClusterInfo was queried and the orchestrator was found
- The endpoint /pools/default was queried
/pools/default/terseClusterInfo response
{u'clusterUUID': u'24734240437d6552109ef98a7b2be95e', u'clusterCompatVersion': u'7.1', u'isBalanced': False, u'orchestrator': u'ns_1@172.23.121.166'}) |
/pools/default returns many fields with "unknown" for the node 172.23.121.169. Also displays status as unhealthy
{u'nodeEncryption': False, u'configuredHostname': u'172.23.121.169:8091', u'externalListeners': [{u'nodeEncryption': False, u'afamily': u'inet'}], u'memoryFree': 0, u'serverGroup': u'Group 1', u'ports': {u'direct': 11210, u'httpsMgmt': 18091, u'distTLS': 21150, u'httpsCAPI': 18092, u'distTCP': 21100}, u'nodeUUID': u'ec8f537a6613304065584729dd8bdbe7', u'recoveryType': u'none', u'hostname': u'172.23.121.169:8091', u'otpNode': u'ns_1@172.23.121.169', u'clusterCompatibility': 458753, u'nodeHash': 32315965, u'addressFamily': u'inet', u'mcdMemoryReserved': 0, u'cpuCount': 0, u'nodeEncryptionClientCertVerification': False, u'os': u'unknown', u'couchApiBase': u'http://172.23.121.169:8092/', u'systemStats': {}, u'mcdMemoryAllocated': 0, u'services': [u'index', u'kv', u'n1ql'], u'version': u'unknown', u'clusterMembership': u'inactiveAdded', u'uptime': u'0', u'interestingStats': {}, u'addressFamilyOnly': False, u'couchApiBaseHTTPS': u'https://172.23.121.169:18092/', u'memoryTotal': 0, u'status': u'unhealthy'} |
After a couple of minutes, the pools/default response contains all expected values for the fields
/pools/default response right after add node - https://cb-engineering.s3.amazonaws.com/MB-60705/pools_default_trinity_unknown.json
/pools/default response after a couple of minutes - https://cb-engineering.s3.amazonaws.com/MB-60705/pools_default_trinity_expected.json
Cluster logs
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063114-ns_1%40172.23.107.53.zip
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063114-ns_1%40172.23.121.166.zip
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063114-ns_1%40172.23.121.169.zip
https://cb-engineering.s3.amazonaws.com/MB-60705/collectinfo-2024-02-07T063114-ns_1%40172.23.123.235.zip
Note that this is only observed when TLS is enabled. Haven't observed this behavior when TLS is disabled. The behavior wasn't caught in 7.2.4 runs as TAF framework was upgraded to enable TLS by default only from Trinity
This issue is affecting many runs in Trinity