Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50694

Initial cluster_init rebalance failed during backup service crash with exit status 1

    XMLWordPrintable

Details

    Description

      Steps:

      Create 111 nodes cluster with service kv, index, n1ql, backup, eventing, fts and perform rebalance

      Observation:

      Rebalance is failing due to the crash in backup service nodes

      Rebalance exited with reason {service_rebalance_failed,backup,
      		 
      {agent_died,<33828.23166.18>,
      		 
      {linked_process_died,<33828.27228.18>,
      		 
      {'ns_1@172.23.122.139',
      		 
      {{badmatch, {false, {topology,[],[],false,[]},
      		 
      {topology,[],
      		 
      [<<"128d0133cfa2051ce2018d5737c3650c">>,
      		 
      <<"3317aecfb62f1c202b7ba6e7934a5bba">>,
      		 
      <<"936988edf0867574b92eebcd5e24ce63">>,
      		 
      <<"a1381ef18ed7ee4fe564dac84d887348">>,
      		 
      <<"ce53fc32f303cb952c613521aaf38652">>,
      		 
      <<"fd3ec752e60f2cd0f677b71813e927ad">>],
      		 
      false,[]}}},
      		 
      [{service_agent,long_poll_worker_loop,5, [{file,"src/service_agent.erl"}, {line,605}]},
      		 
      {proc_lib,init_p,3, [{file,"proc_lib.erl"},{line,211}]}]}}}}}.
      		 
      Rebalance Operation Id = e75d5d44c00511111e321c1becff3439

      Backup services crashes observed on multiple backup nodes

      172.23.122.139:
      		 
       
      		 
      Service 'backup' exited with status 1. Restarting. Messages:
      		 
      2022-01-31T01:54:19.191-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools'
      		 
      2022-01-31T01:54:19.191-08:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default'
      		 
      2022-01-31T01:54:19.685-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default'
      		 
      2022-01-31T01:54:19.688-08:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/buckets'
      		 
      2022-01-31T01:54:19.705-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/buckets'
      		 
      2022-01-31T01:54:19.706-08:00 INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"dd29fe96cdd37eb871310acb50f5dee3","developer_preview":false,"version":{"min_version":"7.1.0","is_mixed_cluster":false},"max_vbuckets":0,"uniform_vbuckets":true}
      		 
      2022-01-31T01:54:19.706-08:00 INFO (Node) Confirmed cluster uuid {"clusterUUID": "dd29fe96cdd37eb871310acb50f5dee3"}
      		 
      2022-01-31T01:54:20.188-08:00 ERROR (Main) Failed to run node {"err": "could not get service configuration: Rev mismatch"}
      		 
       
      		 
      172.23.122.144:
      		 
       
      		 
      Service 'backup' exited with status 1. Restarting. Messages:
      		 
      2022-01-31T01:54:19.160-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools'
      		 
      2022-01-31T01:54:19.160-08:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default'
      		 
      2022-01-31T01:54:19.623-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default'
      		 
      2022-01-31T01:54:19.627-08:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/buckets'
      		 
      2022-01-31T01:54:19.751-08:00 DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/buckets'
      		 
      2022-01-31T01:54:19.751-08:00 INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"dd29fe96cdd37eb871310acb50f5dee3","developer_preview":false,"version":{"min_version":"7.1.0","is_mixed_cluster":false},"max_vbuckets":0,"uniform_vbuckets":true}
      		 
      2022-01-31T01:54:19.751-08:00 INFO (Node) Confirmed cluster uuid {"clusterUUID": "dd29fe96cdd37eb871310acb50f5dee3"}
      		 
      2022-01-31T01:54:19.959-08:00 ERROR (Main) Failed to run node {"err": "could not get service configuration: Rev mismatch"}

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-50560 Failed to run node {"err": "could not get service configuration: Rev mismatch"}

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty