Major Description
K8S Cluster: GKE, v1.20
Steps to Reproduce:
- Install 2.2.1 CRD
- Install 2.2.1 DAC
- Install 2.2.1 Operator
- Create a 2 node Couchbase Cluster running server version 7.0.1-6102 with all services enabled.
- Create a sample bucket `default` on the Cluster.
- Load 10000 docs in the bucket using pillowfight loader.
- Create FTS index with name as `test1` and bucket as `default`, In Advanced Settings change value of Default Analyzer from `Standard` to `keyword`
- Run this Flex Query against query context:
select meta().id from `default` USE INDEX(USING FTS);
Query Result Outputs:
[{"code": 4000,
"msg": "No index available on keyspace `default`:`default` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default` to create a primary index, or check that your expected index is online.",
"query": "select meta().id from `default` USE INDEX(USING FTS);"
}]cbopinfo and Cluster Logs attached.
https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0000.cb-example.default.svc.zip
https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0001.cb-example.default.svc.zip
ns_server.error.log:
==============================================================================
|
couchbase logs (error.log)
|
cbbrowse_logs error.log
|
==============================================================================
|
[ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed", |
{path,
|
"/_metakv/V2InternalSettings/InternalSettings"}, |
{method,'GET'}, |
{type,error},
|
{what,function_clause},
|
{trace,
|
[{misc,compare_secure,
|
[undefined,
|
"2264c032018ae6b06a3e1f28c58fe6d1"], |
[{file,"src/misc.erl"},{line,2927}]}, |
{ns_config_auth,authenticate_special,2, |
[{file,"src/ns_config_auth.erl"}, |
{line,94}]}, |
{ns_config_auth,authenticate,2, |
[{file,"src/ns_config_auth.erl"}, |
{line,104}]}, |
{menelaus_auth,authenticate,1, |
[{file,"src/menelaus_auth.erl"}, |
{line,300}]}, |
{menelaus_auth,verify_rest_auth,2, |
[{file,"src/menelaus_auth.erl"}, |
{line,409}]}, |
{menelaus_web,perform_action,2, |
[{file,"src/menelaus_web.erl"}, |
{line,1124}]}, |
{request_throttler,do_request,3, |
[{file,"src/request_throttler.erl"}, |
{line,58}]}, |
{menelaus_util,handle_request,2, |
[{file,"src/menelaus_util.erl"}, |
{line,216}]}]}] |
[ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper_worker<0.596.0>:rest_utils:get_json:57]Request to (indexer) getIndexStatus with headers [{"If-None-Match", |
"e6789f9a13a311f7"}] failed: {ok, |
{{500, |
"Internal Server Error"}, |
[{"Content-Length", |
"146"}, |
{"Date", |
"Tue, 31 Aug 2021 14:18:59 GMT"}, |
{"Content-Type", |
"application/json"}], |
<<"{\"code\":\"error\",\"error\":\"Fail to retrieve cluster-wide metadata from index service\",\"failedNodes\":[\"cb-example-0001.cb-example.default.svc:8091\"]}">>}} |
[ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper-index<0.608.0>:service_status_keeper:handle_cast:103]Service service_index returned incorrect status |
ns_server.debug.log:
========================NOTICE REPORT=========================
|
{net_kernel,{'EXIT',<0.764.0>,{recv_challenge_reply_failed,{error,closed}}}} |
[ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed", |
{path,
|
"/_metakv/V2InternalSettings/InternalSettings"}, |
{method,'GET'}, |
{type,error},
|
{what,function_clause},
|
{trace,
|
[{misc,compare_secure,
|
[undefined,
|
"2264c032018ae6b06a3e1f28c58fe6d1"], |
[{file,"src/misc.erl"},{line,2927}]}, |
{ns_config_auth,authenticate_special,2, |
[{file,"src/ns_config_auth.erl"}, |
{line,94}]}, |
{ns_config_auth,authenticate,2, |
[{file,"src/ns_config_auth.erl"}, |
{line,104}]}, |
{menelaus_auth,authenticate,1, |
[{file,"src/menelaus_auth.erl"}, |
{line,300}]}, |
{menelaus_auth,verify_rest_auth,2, |
[{file,"src/menelaus_auth.erl"}, |
{line,409}]}, |
{menelaus_web,perform_action,2, |
[{file,"src/menelaus_web.erl"}, |
{line,1124}]}, |
{request_throttler,do_request,3, |
[{file,"src/request_throttler.erl"}, |
{line,58}]}, |
{menelaus_util,handle_request,2, |
[{file,"src/menelaus_util.erl"}, |
{line,216}]}]}] |
[ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Accepted new connection from <0.762.0> DistCtrl #Port<0.65>: {con, |
#Ref<0.3455796189.3111124994.78127>, |
inet_tcp_dist,
|
undefined,
|
undefined}
|
[ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:cb_dist:info_msg:790]cb_dist: Accepting connection from acceptor <0.762.0> using module inet_tcp_dist |
[ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Updated connection: {con,#Ref<0.3455796189.3111124994.78127>, |
inet_tcp_dist,<0.766.0>, |
#Ref<0.3455796189.3111124995.78485>} |
[ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:bringup:267]Attempted to save node name to disk: ok |
[ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:wait_for_node:274]Waiting for connection to node 'babysitter_of_ns_1@cb.local' to be established |
[error_logger:info,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:ale_error_logger_handler:do_log:101] |
Even though error says nodes failed over, Screenshots attached to highlight both server nodes are available and server logs can be collected through them.
Memory Quotas Screenshot attached.
P.S. : The Cluster was accessed using port-forward to create FTS Index.
