Uploaded image for project: 'Couchbase Kubernetes'
  1. Couchbase Kubernetes
  2. K8S-2385

[FTS] Flex Query against FTS Index returns "No Index Available".

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not a Bug
    • 2.2.1
    • None
    • operator, testing
    • None
    • 1

    Description

      K8S Cluster: GKE, v1.20

      Steps to Reproduce:

      1. Install 2.2.1 CRD
      2. Install 2.2.1 DAC
      3. Install 2.2.1 Operator
      4. Create a 2 node Couchbase Cluster running server version 7.0.1-6102 with all services enabled.
      5. Create a sample bucket `default` on the Cluster.
      6. Load 10000 docs in the bucket using pillowfight loader.
      7. Create FTS index with name as `test1` and bucket as `default`, In Advanced Settings change value of Default Analyzer  from `Standard` to `keyword`
      8. Run this Flex Query against query context:

        select meta().id from `default` USE INDEX(USING FTS); 

        Query Result Outputs:

        [
          {
            "code": 4000,
            "msg": "No index available on keyspace `default`:`default` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default` to create a primary index, or check that your expected index is online.",
            "query": "select meta().id from `default` USE INDEX(USING FTS);"
          }
        ]

        cbopinfo and Cluster Logs attached.

      https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0000.cb-example.default.svc.zip
      https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0001.cb-example.default.svc.zip
      ns_server.error.log:

       ==============================================================================
      couchbase logs (error.log)
      cbbrowse_logs error.log
      ==============================================================================
      [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                       {path,
                                        "/_metakv/V2InternalSettings/InternalSettings"},
                                       {method,'GET'},
                                       {type,error},
                                       {what,function_clause},
                                       {trace,
                                        [{misc,compare_secure,
                                          [undefined,
                                           "2264c032018ae6b06a3e1f28c58fe6d1"],
                                          [{file,"src/misc.erl"},{line,2927}]},
                                         {ns_config_auth,authenticate_special,2,
                                          [{file,"src/ns_config_auth.erl"},
                                           {line,94}]},
                                         {ns_config_auth,authenticate,2,
                                          [{file,"src/ns_config_auth.erl"},
                                           {line,104}]},
                                         {menelaus_auth,authenticate,1,
                                          [{file,"src/menelaus_auth.erl"},
                                           {line,300}]},
                                         {menelaus_auth,verify_rest_auth,2,
                                          [{file,"src/menelaus_auth.erl"},
                                           {line,409}]},
                                         {menelaus_web,perform_action,2,
                                          [{file,"src/menelaus_web.erl"},
                                           {line,1124}]},
                                         {request_throttler,do_request,3,
                                          [{file,"src/request_throttler.erl"},
                                           {line,58}]},
                                         {menelaus_util,handle_request,2,
                                          [{file,"src/menelaus_util.erl"},
                                           {line,216}]}]}]
      [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper_worker<0.596.0>:rest_utils:get_json:57]Request to (indexer) getIndexStatus with headers [{"If-None-Match",
                                                         "e6789f9a13a311f7"}] failed: {ok,
                                                                                       {{500,
                                                                                         "Internal Server Error"},
                                                                                        [{"Content-Length",
                                                                                          "146"},
                                                                                         {"Date",
                                                                                          "Tue, 31 Aug 2021 14:18:59 GMT"},
                                                                                         {"Content-Type",
                                                                                          "application/json"}],
                                                                                        <<"{\"code\":\"error\",\"error\":\"Fail to retrieve cluster-wide metadata from index service\",\"failedNodes\":[\"cb-example-0001.cb-example.default.svc:8091\"]}">>}}
      [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper-index<0.608.0>:service_status_keeper:handle_cast:103]Service service_index returned incorrect status
      

      ns_server.debug.log:

      ========================NOTICE REPORT=========================
      {net_kernel,{'EXIT',<0.764.0>,{recv_challenge_reply_failed,{error,closed}}}}
      [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                       {path,
                                        "/_metakv/V2InternalSettings/InternalSettings"},
                                       {method,'GET'},
                                       {type,error},
                                       {what,function_clause},
                                       {trace,
                                        [{misc,compare_secure,
                                          [undefined,
                                           "2264c032018ae6b06a3e1f28c58fe6d1"],
                                          [{file,"src/misc.erl"},{line,2927}]},
                                         {ns_config_auth,authenticate_special,2,
                                          [{file,"src/ns_config_auth.erl"},
                                           {line,94}]},
                                         {ns_config_auth,authenticate,2,
                                          [{file,"src/ns_config_auth.erl"},
                                           {line,104}]},
                                         {menelaus_auth,authenticate,1,
                                          [{file,"src/menelaus_auth.erl"},
                                           {line,300}]},
                                         {menelaus_auth,verify_rest_auth,2,
                                          [{file,"src/menelaus_auth.erl"},
                                           {line,409}]},
                                         {menelaus_web,perform_action,2,
                                          [{file,"src/menelaus_web.erl"},
                                           {line,1124}]},
                                         {request_throttler,do_request,3,
                                          [{file,"src/request_throttler.erl"},
                                           {line,58}]},
                                         {menelaus_util,handle_request,2,
                                          [{file,"src/menelaus_util.erl"},
                                           {line,216}]}]}]
      [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Accepted new connection from <0.762.0> DistCtrl #Port<0.65>: {con,
                                                                             #Ref<0.3455796189.3111124994.78127>,
                                                                             inet_tcp_dist,
                                                                             undefined,
                                                                             undefined}
      [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:cb_dist:info_msg:790]cb_dist: Accepting connection from acceptor <0.762.0> using module inet_tcp_dist
      [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Updated connection: {con,#Ref<0.3455796189.3111124994.78127>,
                                        inet_tcp_dist,<0.766.0>,
                                        #Ref<0.3455796189.3111124995.78485>}
      [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:bringup:267]Attempted to save node name to disk: ok
      [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:wait_for_node:274]Waiting for connection to node 'babysitter_of_ns_1@cb.local' to be established
      [error_logger:info,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:ale_error_logger_handler:do_log:101] 

      Even though error says nodes failed over, Screenshots attached to highlight both server nodes are available and server logs can be collected through them.

      Memory Quotas Screenshot attached.

      P.S. : The Cluster was accessed using port-forward to create FTS Index.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          prateek.kumar Prateek Kumar created issue -
          prateek.kumar Prateek Kumar made changes -
          Field Original Value New Value
          Description K8S Cluster: GKE, v1.20

          Steps to Reproduce:
           # Install 2.2.1 CRD
           # Install 2.2.1 DAC
           # Install 2.2.1 Operator
           # Create a 2 node Couchbase Cluster running server version 7.0.1-6102 with all services enabled.
           # Create a sample bucket `default` on the Cluster.
           # Load 10000 docs in the bucket using pillowfight loader.
           # Create FTS index with name as `test1` and bucket as `default`, In Advanced Settings change value of Default Analyzer  from `Standard` to `keyword`
           # Run this Flex Query against query context:
          {code:java}
          select meta().id from `default` USE INDEX(USING FTS); {code}

           # Query Result Outputs:
          {code:java}
          [
            {
              "code": 4000,
              "msg": "No index available on keyspace `default`:`travel-sample` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`travel-sample` to create a primary index, or check that your expected index is online.",
              "query": "select meta().id from `travel-sample` USE INDEX(USING FTS);"
            }
          ] {code}
          cbopinfo and Cluster Logs attached.

          [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0000.cb-example.default.svc.zip]
          [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0001.cb-example.default.svc.zip]
          ns_server.error.log:
          {code:java}
           ==============================================================================
          couchbase logs (error.log)
          cbbrowse_logs error.log
          ==============================================================================
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper_worker<0.596.0>:rest_utils:get_json:57]Request to (indexer) getIndexStatus with headers [{"If-None-Match",
                                                             "e6789f9a13a311f7"}] failed: {ok,
                                                                                           {{500,
                                                                                             "Internal Server Error"},
                                                                                            [{"Content-Length",
                                                                                              "146"},
                                                                                             {"Date",
                                                                                              "Tue, 31 Aug 2021 14:18:59 GMT"},
                                                                                             {"Content-Type",
                                                                                              "application/json"}],
                                                                                            <<"{\"code\":\"error\",\"error\":\"Fail to retrieve cluster-wide metadata from index service\",\"failedNodes\":[\"cb-example-0001.cb-example.default.svc:8091\"]}">>}}
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper-index<0.608.0>:service_status_keeper:handle_cast:103]Service service_index returned incorrect status
          {code}
          ns_server.debug.log:
          {code:java}
          ========================NOTICE REPORT=========================
          {net_kernel,{'EXIT',<0.764.0>,{recv_challenge_reply_failed,{error,closed}}}}
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Accepted new connection from <0.762.0> DistCtrl #Port<0.65>: {con,
                                                                                 #Ref<0.3455796189.3111124994.78127>,
                                                                                 inet_tcp_dist,
                                                                                 undefined,
                                                                                 undefined}
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:cb_dist:info_msg:790]cb_dist: Accepting connection from acceptor <0.762.0> using module inet_tcp_dist
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Updated connection: {con,#Ref<0.3455796189.3111124994.78127>,
                                            inet_tcp_dist,<0.766.0>,
                                            #Ref<0.3455796189.3111124995.78485>}
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:bringup:267]Attempted to save node name to disk: ok
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:wait_for_node:274]Waiting for connection to node 'babysitter_of_ns_1@cb.local' to be established
          [error_logger:info,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:ale_error_logger_handler:do_log:101] {code}
          Even though error says nodes failed over, Screenshots attached to highlight both server nodes are available and server logs can be collected through them.

          Memory Quotas Screenshot attached.

          P.S. : The Cluster was accessed using port-forward to create FTS Index.
          K8S Cluster: GKE, v1.20

          Steps to Reproduce:
           # Install 2.2.1 CRD
           # Install 2.2.1 DAC
           # Install 2.2.1 Operator
           # Create a 2 node Couchbase Cluster running server version 7.0.1-6102 with all services enabled.
           # Create a sample bucket `default` on the Cluster.
           # Load 10000 docs in the bucket using pillowfight loader.
           # Create FTS index with name as `test1` and bucket as `default`, In Advanced Settings change value of Default Analyzer  from `Standard` to `keyword`
           # Run this Flex Query against query context:
          {code:java}select meta().id from `default` USE INDEX(USING FTS); {code}Query Result Outputs:
          {code:java}[
            {
              "code": 4000,
              "msg": "No index available on keyspace `default`:`travel-sample` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`travel-sample` to create a primary index, or check that your expected index is online.",
              "query": "select meta().id from `travel-sample` USE INDEX(USING FTS);"
            }
          ] {code}
          cbopinfo and Cluster Logs attached.

          [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0000.cb-example.default.svc.zip]
           [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0001.cb-example.default.svc.zip]
           ns_server.error.log:
          {code:java} ==============================================================================
          couchbase logs (error.log)
          cbbrowse_logs error.log
          ==============================================================================
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper_worker<0.596.0>:rest_utils:get_json:57]Request to (indexer) getIndexStatus with headers [{"If-None-Match",
                                                             "e6789f9a13a311f7"}] failed: {ok,
                                                                                           {{500,
                                                                                             "Internal Server Error"},
                                                                                            [{"Content-Length",
                                                                                              "146"},
                                                                                             {"Date",
                                                                                              "Tue, 31 Aug 2021 14:18:59 GMT"},
                                                                                             {"Content-Type",
                                                                                              "application/json"}],
                                                                                            <<"{\"code\":\"error\",\"error\":\"Fail to retrieve cluster-wide metadata from index service\",\"failedNodes\":[\"cb-example-0001.cb-example.default.svc:8091\"]}">>}}
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper-index<0.608.0>:service_status_keeper:handle_cast:103]Service service_index returned incorrect status
          {code}
          ns_server.debug.log:
          {code:java}========================NOTICE REPORT=========================
          {net_kernel,{'EXIT',<0.764.0>,{recv_challenge_reply_failed,{error,closed}}}}
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Accepted new connection from <0.762.0> DistCtrl #Port<0.65>: {con,
                                                                                 #Ref<0.3455796189.3111124994.78127>,
                                                                                 inet_tcp_dist,
                                                                                 undefined,
                                                                                 undefined}
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:cb_dist:info_msg:790]cb_dist: Accepting connection from acceptor <0.762.0> using module inet_tcp_dist
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Updated connection: {con,#Ref<0.3455796189.3111124994.78127>,
                                            inet_tcp_dist,<0.766.0>,
                                            #Ref<0.3455796189.3111124995.78485>}
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:bringup:267]Attempted to save node name to disk: ok
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:wait_for_node:274]Waiting for connection to node 'babysitter_of_ns_1@cb.local' to be established
          [error_logger:info,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:ale_error_logger_handler:do_log:101] {code}
          Even though error says nodes failed over, Screenshots attached to highlight both server nodes are available and server logs can be collected through them.

          Memory Quotas Screenshot attached.

          P.S. : The Cluster was accessed using port-forward to create FTS Index.
          prateek.kumar Prateek Kumar made changes -
          Description K8S Cluster: GKE, v1.20

          Steps to Reproduce:
           # Install 2.2.1 CRD
           # Install 2.2.1 DAC
           # Install 2.2.1 Operator
           # Create a 2 node Couchbase Cluster running server version 7.0.1-6102 with all services enabled.
           # Create a sample bucket `default` on the Cluster.
           # Load 10000 docs in the bucket using pillowfight loader.
           # Create FTS index with name as `test1` and bucket as `default`, In Advanced Settings change value of Default Analyzer  from `Standard` to `keyword`
           # Run this Flex Query against query context:
          {code:java}select meta().id from `default` USE INDEX(USING FTS); {code}Query Result Outputs:
          {code:java}[
            {
              "code": 4000,
              "msg": "No index available on keyspace `default`:`travel-sample` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`travel-sample` to create a primary index, or check that your expected index is online.",
              "query": "select meta().id from `travel-sample` USE INDEX(USING FTS);"
            }
          ] {code}
          cbopinfo and Cluster Logs attached.

          [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0000.cb-example.default.svc.zip]
           [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0001.cb-example.default.svc.zip]
           ns_server.error.log:
          {code:java} ==============================================================================
          couchbase logs (error.log)
          cbbrowse_logs error.log
          ==============================================================================
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper_worker<0.596.0>:rest_utils:get_json:57]Request to (indexer) getIndexStatus with headers [{"If-None-Match",
                                                             "e6789f9a13a311f7"}] failed: {ok,
                                                                                           {{500,
                                                                                             "Internal Server Error"},
                                                                                            [{"Content-Length",
                                                                                              "146"},
                                                                                             {"Date",
                                                                                              "Tue, 31 Aug 2021 14:18:59 GMT"},
                                                                                             {"Content-Type",
                                                                                              "application/json"}],
                                                                                            <<"{\"code\":\"error\",\"error\":\"Fail to retrieve cluster-wide metadata from index service\",\"failedNodes\":[\"cb-example-0001.cb-example.default.svc:8091\"]}">>}}
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper-index<0.608.0>:service_status_keeper:handle_cast:103]Service service_index returned incorrect status
          {code}
          ns_server.debug.log:
          {code:java}========================NOTICE REPORT=========================
          {net_kernel,{'EXIT',<0.764.0>,{recv_challenge_reply_failed,{error,closed}}}}
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Accepted new connection from <0.762.0> DistCtrl #Port<0.65>: {con,
                                                                                 #Ref<0.3455796189.3111124994.78127>,
                                                                                 inet_tcp_dist,
                                                                                 undefined,
                                                                                 undefined}
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:cb_dist:info_msg:790]cb_dist: Accepting connection from acceptor <0.762.0> using module inet_tcp_dist
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Updated connection: {con,#Ref<0.3455796189.3111124994.78127>,
                                            inet_tcp_dist,<0.766.0>,
                                            #Ref<0.3455796189.3111124995.78485>}
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:bringup:267]Attempted to save node name to disk: ok
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:wait_for_node:274]Waiting for connection to node 'babysitter_of_ns_1@cb.local' to be established
          [error_logger:info,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:ale_error_logger_handler:do_log:101] {code}
          Even though error says nodes failed over, Screenshots attached to highlight both server nodes are available and server logs can be collected through them.

          Memory Quotas Screenshot attached.

          P.S. : The Cluster was accessed using port-forward to create FTS Index.
          K8S Cluster: GKE, v1.20

          Steps to Reproduce:
           # Install 2.2.1 CRD
           # Install 2.2.1 DAC
           # Install 2.2.1 Operator
           # Create a 2 node Couchbase Cluster running server version 7.0.1-6102 with all services enabled.
           # Create a sample bucket `default` on the Cluster.
           # Load 10000 docs in the bucket using pillowfight loader.
           # Create FTS index with name as `test1` and bucket as `default`, In Advanced Settings change value of Default Analyzer  from `Standard` to `keyword`
           # Run this Flex Query against query context:
          {code:java}select meta().id from `default` USE INDEX(USING FTS); {code}
          Query Result Outputs:
          {code:java}[
            {
              "code": 4000,
              "msg": "No index available on keyspace `default`:`default` that matches your query. Use CREATE PRIMARY INDEX ON `default`:`default` to create a primary index, or check that your expected index is online.",
              "query": "select meta().id from `default` USE INDEX(USING FTS);"
            }
          ]{code}
          cbopinfo and Cluster Logs attached.

          [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0000.cb-example.default.svc.zip]
           [https://cb-engineering.s3.amazonaws.com/Fts/collectinfo-2021-08-31T142309-ns_1%40cb-example-0001.cb-example.default.svc.zip]
           ns_server.error.log:
          {code:java} ==============================================================================
          couchbase logs (error.log)
          cbbrowse_logs error.log
          ==============================================================================
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper_worker<0.596.0>:rest_utils:get_json:57]Request to (indexer) getIndexStatus with headers [{"If-None-Match",
                                                             "e6789f9a13a311f7"}] failed: {ok,
                                                                                           {{500,
                                                                                             "Internal Server Error"},
                                                                                            [{"Content-Length",
                                                                                              "146"},
                                                                                             {"Date",
                                                                                              "Tue, 31 Aug 2021 14:18:59 GMT"},
                                                                                             {"Content-Type",
                                                                                              "application/json"}],
                                                                                            <<"{\"code\":\"error\",\"error\":\"Fail to retrieve cluster-wide metadata from index service\",\"failedNodes\":[\"cb-example-0001.cb-example.default.svc:8091\"]}">>}}
          [ns_server:error,2021-08-31T14:18:59.300Z,ns_1@cb-example-0000.cb-example.default.svc:service_status_keeper-index<0.608.0>:service_status_keeper:handle_cast:103]Service service_index returned incorrect status
          {code}
          ns_server.debug.log:
          {code:java}========================NOTICE REPORT=========================
          {net_kernel,{'EXIT',<0.764.0>,{recv_challenge_reply_failed,{error,closed}}}}
          [ns_server:error,2021-08-31T14:18:34.535Z,ns_1@cb-example-0000.cb-example.default.svc:<0.499.0>:menelaus_util:reply_server_error:205]Server error during processing: ["web request failed",
                                           {path,
                                            "/_metakv/V2InternalSettings/InternalSettings"},
                                           {method,'GET'},
                                           {type,error},
                                           {what,function_clause},
                                           {trace,
                                            [{misc,compare_secure,
                                              [undefined,
                                               "2264c032018ae6b06a3e1f28c58fe6d1"],
                                              [{file,"src/misc.erl"},{line,2927}]},
                                             {ns_config_auth,authenticate_special,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,94}]},
                                             {ns_config_auth,authenticate,2,
                                              [{file,"src/ns_config_auth.erl"},
                                               {line,104}]},
                                             {menelaus_auth,authenticate,1,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,300}]},
                                             {menelaus_auth,verify_rest_auth,2,
                                              [{file,"src/menelaus_auth.erl"},
                                               {line,409}]},
                                             {menelaus_web,perform_action,2,
                                              [{file,"src/menelaus_web.erl"},
                                               {line,1124}]},
                                             {request_throttler,do_request,3,
                                              [{file,"src/request_throttler.erl"},
                                               {line,58}]},
                                             {menelaus_util,handle_request,2,
                                              [{file,"src/menelaus_util.erl"},
                                               {line,216}]}]}]
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Accepted new connection from <0.762.0> DistCtrl #Port<0.65>: {con,
                                                                                 #Ref<0.3455796189.3111124994.78127>,
                                                                                 inet_tcp_dist,
                                                                                 undefined,
                                                                                 undefined}
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:cb_dist:info_msg:790]cb_dist: Accepting connection from acceptor <0.762.0> using module inet_tcp_dist
          [ns_server:debug,2021-08-31T14:18:34.538Z,ns_1@cb-example-0000.cb-example.default.svc:cb_dist<0.758.0>:cb_dist:info_msg:790]cb_dist: Updated connection: {con,#Ref<0.3455796189.3111124994.78127>,
                                            inet_tcp_dist,<0.766.0>,
                                            #Ref<0.3455796189.3111124995.78485>}
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:bringup:267]Attempted to save node name to disk: ok
          [ns_server:debug,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:<0.747.0>:dist_manager:wait_for_node:274]Waiting for connection to node 'babysitter_of_ns_1@cb.local' to be established
          [error_logger:info,2021-08-31T14:18:34.539Z,ns_1@cb-example-0000.cb-example.default.svc:net_kernel<0.761.0>:ale_error_logger_handler:do_log:101] {code}
          Even though error says nodes failed over, Screenshots attached to highlight both server nodes are available and server logs can be collected through them.

          Memory Quotas Screenshot attached.

          P.S. : The Cluster was accessed using port-forward to create FTS Index.
          prateek.kumar Prateek Kumar made changes -
          Assignee Prateek Kumar [ prateek.kumar ] Simon Murray [ simon.murray ]
          prateek.kumar Prateek Kumar made changes -
          prateek.kumar Prateek Kumar made changes -
          Assignee Simon Murray [ simon.murray ] Prateek Kumar [ prateek.kumar ]
          Resolution Not a Bug [ 10200 ]
          Status Open [ 1 ] Closed [ 6 ]

          People

            prateek.kumar Prateek Kumar
            prateek.kumar Prateek Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty