Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60347

Rebalance exited with reason {mover_crashed, {{error, {badrpc, {'EXIT', {{{{badmatch,{ok,[]}},

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 7.6.0
    • 7.6.0
    • view-engine
    • None
    • Enterprise Edition 7.6.0 build 1980

    Description

      Steps:
      1. Create a 2 node cluster

      [ns_server:info,2024-01-11T02:45:10.359-08:00,ns_1@172.23.105.191:<0.10883.0>:ns_orchestrator:idle:927]Starting rebalance, KeepNodes = ['ns_1@172.23.104.235','ns_1@172.23.105.191'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = d9cc8a280b450643f7001e778e7f2790
      [user:info,2024-01-11T02:45:10.360-08:00,ns_1@172.23.105.191:<0.10883.0>:ns_orchestrator:idle:930]Starting rebalance, KeepNodes = ['ns_1@172.23.104.235','ns_1@172.23.105.191'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = d9cc8a280b450643f7001e778e7f2790
      [ns_server:info,2024-01-11T02:45:11.611-08:00,ns_1@172.23.105.191:<0.10883.0>:ns_orchestrator:handle_event:670]Skipping janitor in s

      2. creating a bucket 'bucket-0'

      [menelaus:info,2024-01-11T02:45:44.076-08:00,ns_1@172.23.105.191:<0.12857.0>:menelaus_web_buckets:do_bucket_create:877]Created bucket "bucket-0" of type: couchbase
      [{num_replicas,1},
       {replica_index,true},
       {ram_quota,268435456},
       {durability_min_level,none},
       {storage_quota_percentage,50},
       {num_vbuckets,1024},
       {cross_cluster_versioning_enabled,false},
       {version_pruning_window_hrs,720},
       {pitr_enabled,false},
       {pitr_granularity,600},
       {pitr_max_history_age,86400},
       {autocompaction,false},
       {purge_interval,undefined},
       {flush_enabled,false},
       {num_threads,3},
       {eviction_policy,full_eviction},
       {conflict_resolution_type,seqno},
       {storage_mode,magma},
       {max_ttl,0},
       {compression_mode,passive},
       {history_retention_seconds,0},
       {history_retention_bytes,0},
       {magma_key_tree_data_blocksize,4096},
       {magma_seq_tree_data_blocksize,4096},
       {history_retention_collection_default,true},
       {rank,876}]
      

      3. creating travel-sample and beer-sample from sample buckets

      4. Attempting to rebalance in a node(172.23.104.234) post creating sample buckets caused this failure

      [user:error,2024-01-11T09:33:27.289-08:00,ns_1@172.23.105.191:<0.10883.0>:ns_orchestrator:log_rebalance_completion:1661]Rebalance exited with reason {mover_crashed,
                                    {{error,
                                      {badrpc,
                                       {'EXIT',
                                        {{{{badmatch,{ok,[]}},
                                           [{couch_set_view_group,
                                             process_monitor_partition_update,4,
                                             [{file,
                                               "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},
                                              {line,3771}]},
                                            {couch_set_view_group,handle_call,3,
                                             [{file,
                                               "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},
                                              {line,935}]},
                                            {gen_server,try_handle_call,4,
                                             [{file,"gen_server.erl"},{line,1149}]},
                                            {gen_server,handle_msg,6,
                                             [{file,"gen_server.erl"},{line,1178}]},
                                            {proc_lib,init_p_do_apply,3,
                                             [{file,"proc_lib.erl"},{line,240}]}]},
                                          {gen_server,call,
                                           [<17220.3575.0>,
                                            {monitor_partition_update,1020,
                                             #Ref<17220.656486219.3806330882.217925>,
                                             <17220.3626.0>},
                                            infinity]}},
                                         {gen_server,call,
                                          ['capi_set_view_manager-beer-sample',
                                           {wait_index_updated,1020},
                                           infinity]}}}}},
                                     {gen_server,call,
                                      [{'janitor_agent-beer-sample',
                                        'ns_1@172.23.104.234'},
                                       {if_rebalance
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pulkit.matta Pulkit Matta
            pulkit.matta Pulkit Matta
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty