Critical Description
As part of MB-26756, Eventing switched to using system RBAC user for doing bucket/n1ql ops using LCB/GoCB. Design flow of this implementation is straightforward:
- LCB/GoCB invoke a callback function with host and port combination of KV node that they need auth for in-order to perform bucket/n1ql op.
- When Eventing receives auth request for KV host:port combination it calls cbauth.GetMemcachedServiceAuth for supplied host:port
===
Now the issue that's observed:
- Both LCB and GoCB request for auth against 10.111.170.102:11210 but cbauth refused the request with below error:
logs/eventing.log:2018-02-27T03:20:30.185-08:00 [Error] UTIL GoCB Failed to get credentials for endpoint: <ud>10.111.170.102:11210</ud>, err: Unable to find given hostport in cbauth database: `10.111.170.102:11210'
|
- Looking at the debug dump from LCB around the same time:
2018-02-27T03:20:30.573-08:00 [INFO] === CLUSTER CONFIG BEGIN ===
|
2018-02-27T03:20:30.573-08:00 [INFO] {"nodeLocator":"vbucket","uuid":"840cc1ae009e3bda125cf7c621be74a8","rev":58,"name":"metadata","nodesExt":[{"hostname":"10.111.170.101","services":{"mgmt":8091,"capi":8092,"kv":11210,"mgmtSSL":18091,"capiSSL":18092,"kvSSL":11207}},{"hostname":"10.111.170.102","services":{"mgmt":8091,"capi":8092,"kv":11210,"mgmtSSL":18091,"capiSSL":18092,"kvSSL":11207}},{"hostname":"10.111.170.103","services":{"mgmt":8091,"capi":8092,"kv":11210,"mgmtSSL":18091,"capiSSL":18092,"kvSSL":11207}},{"hostname":"10.111.170.104","services":{"mgmt":8091,"mgmtSSL":18091}},{"hostname":"10.112.170.101","services":{"mgmt":8091,"mgmtSSL":18091}},{"hostname":"10.112.170.102","services":{"mgmt":8091,"mgmtSSL":18091}},{"hostname":"10.112.170.103","services":{"mgmt":8091,"n1ql":8093,"indexScan":9101,"indexAdmin":9100,"mgmtSSL":18091,"n1qlSSL":18093}}],
|
From above cluster config dump it's clear that 10.111.170.102:11210 is valid combination to request auth for but cbauth errors out.
{"hostname":"10.111.170.102","services":{"mgmt":8091,"capi":8092,"kv":11210,"mgmtSSL":18091,"capiSSL":18092,"kvSSL":11207}},
|
Raw couchbase logs from the run are available here: s3://cb-customers-secure/couchbase/toy_build_run.zip
Issue can be replicated on cluster_run setup as well(would require to setup testrunner - need to spin up cluster_run -n8, if you need .ini file - could comment here):
testrunner.py -i ~/.eventing.ini GROUP=bucket_op_with_timers -t eventing.eventing_rebalance.EventingRebalance.test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations,nodes_init=7,services_init=kv-kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,reset_services=True,doc-per-day=10,handler_code=bucket_op_with_timers,GROUP=bucket_op_with_timers
|
