Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49960

CE: initial_rebalance_failed,index: serverGroups API: HTTP error 400 Bad Request

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1831

      Install 7.1.0-1831 CE. We see below error in the logs.

      [ns_server:error,2021-12-07T12:58:20.160-08:00,ns_1@cb.local:cleanup_process<0.1270.0>:service_janitor:init_topology_aware_service:91]Initial rebalance for `index` failed: {error,
                                             {initial_rebalance_failed,index,
                                              {agent_died,<0.1173.0>,
                                               {linked_process_died,<0.1174.0>,
                                                {'ns_1@cb.local',
                                                 {no_connection,
                                                  "index-service_api"}}}}}}
      [ns_server:debug,2021-12-07T12:58:20.160-08:00,ns_1@cb.local:cleanup_process<0.1270.0>:service_janitor:init_topology_aware_service:84]Doing initial topology change for service `n1ql'
      [ns_server:debug,2021-12-07T12:58:20.160-08:00,ns_1@cb.local:service_rebalancer-n1ql<0.2698.0>:service_agent:wait_for_agents:69]Waiting for the service agents for service n1ql to come up on nodes:
      ['ns_1@cb.local']
      [ns_server:debug,2021-12-07T12:58:20.161-08:00,ns_1@cb.local:service_rebalancer-n1ql<0.2698.0>:service_agent:wait_for_agents_loop:87]All service agents are ready for n1ql
      [rebalance:info,2021-12-07T12:58:20.161-08:00,ns_1@cb.local:service_rebalancer-n1ql-worker<0.2708.0>:service_rebalancer:rebalance_worker:147]Rebalancing service n1ql with id <<"a5a93fb4ffc42c075cbded104c08aee7">>.
      KeepNodes: ['ns_1@cb.local']
      EjectNodes: []
      DeltaNodes: []
      [ns_server:debug,2021-12-07T12:58:20.161-08:00,ns_1@cb.local:menelaus_cbauth<0.899.0>:menelaus_cbauth:handle_cast:101]Observed json rpc process {"goxdcr-cbauth",<0.647.0>} needs_update
      [ns_server:debug,2021-12-07T12:58:20.162-08:00,ns_1@cb.local:service_rebalancer-n1ql-worker<0.2708.0>:service_rebalancer:rebalance_worker:153]Got node infos:
      [{'ns_1@cb.local',[{node_id,<<"b573c750f484a18e76bfdb7775084b99">>},
                         {priority,0},
                         {opaque,null}]}]
      [ns_server:debug,2021-12-07T12:58:20.162-08:00,ns_1@cb.local:menelaus_cbauth<0.899.0>:menelaus_cbauth:handle_cast:101]Observed json rpc process {"projector-cbauth",<0.1196.0>} needs_update
      [ns_server:debug,2021-12-07T12:58:20.163-08:00,ns_1@cb.local:menelaus_cbauth<0.899.0>:menelaus_cbauth:handle_cast:101]Observed json rpc process {"cbq-engine-cbauth",<0.1200.0>} needs_update
      [ns_server:info,2021-12-07T12:58:23.767-08:00,ns_1@cb.local:<0.1014.0>:ns_orchestrator:handle_event:492]Skipping janitor in state janitor_running
      [ns_server:error,2021-12-07T12:58:23.774-08:00,ns_1@cb.local:service_status_keeper_worker<0.963.0>:rest_utils:get_json:62]Request to (indexer) getIndexStatus with headers [] failed: {error,
                                                                   {econnrefused,
                                                                    [{lhttpc_client,
                                                                      send_request,
                                                                      1,
                                                                      [{file,
                                                                        "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
                                                                       {line,220}]},
                                                                     {lhttpc_client,
                                                                      execute,9,
                                                                      [{file,
                                                                        "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
                                                                       {line,169}]},
                                                                     {lhttpc_client,
                                                                      request,9,
                                                                      [{file,
                                                                        "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
                                                                       {line,
                                                                        93}]}]}}
      
      

      On index log:

      2021-12-07T12:57:20.321-08:00 [Warn] certifcate location is missing.  Cannot refresh certifcate
      2021-12-07T12:57:20.321-08:00 [Info] Encryption config refresh successfully.   Encryption enabled=false
      2021-12-07T12:57:20.321-08:00 [Info] security context:  encryptLocalHost true Local IP's: map[127.0.0.1:true 172.23.97.211:true ::1:true fd63:6f75:6368:2060:70de:93ff:fe0f:a193:true fe80::70de:93ff:fe0f:a193:true]
      2021-12-07T12:57:22.331-08:00 [Info] Error occurred during cluster info update (HTTP error 400 Bad Request getting "http://127.0.0.1:8091/pools/default/serverGroups?v=86973393": This http API endpoint requires enterprise edition) .. Retrying(1)
      2021-12-07T12:57:24.336-08:00 [Info] Error occurred during cluster info update (HTTP error 400 Bad Request getting "http://127.0.0.1:8091/pools/default/serverGroups?v=86973393": This http API endpoint requires enterprise edition) .. Retrying(2)
      2021-12-07T12:57:26.341-08:00 [Info] Error occurred during cluster info update (HTTP error 400 Bad Request getting "http://127.0.0.1:8091/pools/default/serverGroups?v=86973393": This http API endpoint requires enterprise edition) .. Retrying(3)
      2021-12-07T12:57:28.347-08:00 [Info] Error occurred during cluster info update (HTTP error 400 Bad Request getting "http://127.0.0.1:8091/pools/default/serverGroups?v=86973393": This http API endpoint requires enterprise edition) .. Retrying(4)
      2021-12-07T12:57:30.351-08:00 [Info] Error occurred during cluster info update (HTTP error 400 Bad Request getting "http://127.0.0.1:8091/pools/default/serverGroups?v=86973393": This http API endpoint requires enterprise edition) .. Retrying(5)
      2021-12-07T12:57:32.356-08:00 [Info] Error occurred during cl
      

      Could not do cb collect, so attaching the zip
      Logs:

      logs.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sai.teja Sai Krishna Teja
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty