Hareen Kancharla added a comment - 23/Aug/21 9:01 AM Roo Thorp : Quick clarification question, Do you see "rebalance" set to false much after the rebalance has finished? "operator repeatedly tries to rebalance because /pools/default responds with "balanced": false" How frequently and how often is the operator calling the pools/default API?

Roo Thorp added a comment - 23/Aug/21 9:27 AM Hi Hareen Kancharla , Looking at the logs, it looks like when not rebalancing we query pool/default every ~3 seconds, and during a rebalance its every ~1 second. We always see "balanced":false - in the logs I cannot find an instance of this being true.

Hareen Kancharla added a comment - 23/Aug/21 9:40 AM Thanks Roo Thorp . Let me dig into it further.

Hareen Kancharla added a comment - 24/Aug/21 5:22 PM - edited From the logs and code I see: 1) pools/default handler gets the status of all the services from ns_doctor. In ns_doctor logs I see the service_status of n1ql is always "needs_rebalance: true" [ns_doctor:debug,2021-08-20T14:27:57.190Z,ns_1@cb-example-0000.cb-example.default.svc:ns_doctor<0.863.0>:ns_doctor:handle_info:184]Current node statuses: [{'ns_1@cb-example-0000.cb-example.default.svc', {{service_status,n1ql},[{connected,true},{needs_rebalance,true}]}, [ns_doctor:debug,2021-08-20T14:28:57.227Z,ns_1@cb-example-0000.cb-example.default.svc:ns_doctor<0.863.0>:ns_doctor:handle_info:184]Current node statuses: {{service_status,n1ql},[{connected,true},{needs_rebalance,true}]}, [ns_doctor:debug,2021-08-20T14:29:57.230Z,ns_1@cb-example-0000.cb-example.default.svc:ns_doctor<0.863.0>:ns_doctor:handle_info:184]Current node statuses: {{service_status,n1ql},[{connected,true},{needs_rebalance,true}]}, [ns_doctor:debug,2021-08-20T14:31:57.249Z,ns_1@cb-example-0000.cb-example.default.svc:ns_doctor<0.863.0>:ns_doctor:handle_info:184]Current node statuses: {{service_status,n1ql},[{connected,true},{needs_rebalance,true}]}, 2) ns_doctor service_status is updated by the n1ql service_agent which gets incorrect json rpc response for "GetCurrentTopology" from N1QL service. We expect the "nodes" in the response below to be node UUID's, but we receive the node names in the response. [json_rpc:debug,2021-08-20T14:32:44.309Z,ns_1@cb-example-0000.cb-example.default.svc:json_rpc_connection-n1ql-service_api<0.1086.0>:json_rpc_connection:handle_call:152]sending jsonrpc call:{[{jsonrpc,<<"2.0">>}, {id,290}, {method,<<"ServiceAPI.GetCurrentTopology">>}, {params,[{[{rev,null},{timeout,30000}]}]}]} [json_rpc:debug,2021-08-20T14:32:44.312Z,ns_1@cb-example-0000.cb-example.default.svc:json_rpc_connection-n1ql-service_api<0.1086.0>:json_rpc_connection:handle_info:88]got response: [{<<"id">>,290}, {<<"result">>, {[{<<"rev">>,<<"AAAAAAAAABg=">>}, {<<"nodes">>, [<<"cb-example-0000.cb-example.default.svc:8091">>, ##### HK: These should be node UUIDs and not node names. <<"cb-example-0001.cb-example.default.svc:8091">>, <<"cb-example-0002.cb-example.default.svc:8091">>]}, {<<"isBalanced">>,true}]}}, {<<"error">>,null}] [json_rpc:debug,2021-08-20T14:32:44.315Z,ns_1@cb-example-0000.cb-example.default.svc:json_rpc_connection-n1ql-service_api<0.1086.0>:json_rpc_connection:handle_call:152]sending jsonrpc call:{[{jsonrpc,<<"2.0">>}, {id,294}, {method,<<"ServiceAPI.GetCurrentTopology">>}, {params,[{[{rev,null},{timeout,30000}]}]}]} [json_rpc:debug,2021-08-20T14:32:44.316Z,ns_1@cb-example-0000.cb-example.default.svc:json_rpc_connection-n1ql-service_api<0.1086.0>:json_rpc_connection:handle_info:88]got response: [{<<"id">>,294}, {<<"result">>, {[{<<"rev">>,<<"AAAAAAAAABg=">>}, {<<"nodes">>, [<<"cb-example-0000.cb-example.default.svc:8091">>, ###### HK: we expect these to be UUID's. <<"cb-example-0001.cb-example.default.svc:8091">>, <<"cb-example-0002.cb-example.default.svc:8091">>]}, {<<"isBalanced">>,true}]}}, {<<"error">>,null}] Moving the ticket to the N1QL team to take a look at it further.

Couchbase Build Team added a comment - 25/Aug/21 5:06 AM Build couchbase-server-7.1.0-1198 contains query commit 006703e with commit message: MB-48077 Report topology as UUIDs not host names

Donald Haggart added a comment - 25/Aug/21 5:09 AM Should be able to verify by simply repeating the testing using an appropriate build. (couchbase-server-7.1.0-1198 or later)

Roo Thorp added a comment - 25/Aug/21 7:14 AM Hi Donald Haggart , Thanks for this! I'll need to wait a bit for this build to be docker-ized, but when it's available I'll check and update the ticket. Thanks!