Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58217

KV swap rebalance operation failed with CheckpointManager::queueDirty: Got status:failure:duplicate item when vb is non-active

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • None
    • 7.2.1
    • couchbase-bucket
    • Enterprise Edition 7.2.1 build 5904
    • Untriaged
    • Linux x86_64
    • 0
    • Yes

    Description

      QE Test

      ./testrunner -i /tmp/testexec.111655.ini -p get-cbcollect-info=True,reset_services=False,java_sdk_client=True,bucket_storage=magma,GROUP=magma_rebalance_20dgm_set2,get-cbcollect-info=True,enable_cdc=True -t eventing.eventing_rebalance_collection.EventingRebalanceCollection.test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations,nodes_init=6,services_init=kv-kv-eventing-eventing-eventing-index:n1ql,dataset=default,groups=simple,,reset_services=True,bucket_storage=magma,number_of_documents=640000,bucket_size=512,eviction_policy=fullEviction,dgm_run=True,batch_size=100000,java_sdk_client=True,GROUP=magma_rebalance_20dgm_set2
      

      Rebalance Operation

      2023-08-08T09:15:01.154-07:00, ns_orchestrator:0:info:message(ns_1@172.23.122.170) - Starting rebalance, KeepNodes = ['ns_1@172.23.107.111','ns_1@172.23.107.112',
                                       'ns_1@172.23.107.132','ns_1@172.23.107.149',
                                       'ns_1@172.23.108.188','ns_1@172.23.122.170'], EjectNodes = ['ns_1@172.23.122.98'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 469afa0e65ed41917f3cc9c7050f1586
      

      Swap rebalance of pair of KV nodes.

      Rebalance Failure

      2023-08-08T09:37:35.600-07:00, ns_vbucket_mover:0:critical:message(ns_1@172.23.122.170) - Worker <0.6496.22> (for action {move,{1023,
                                            ['ns_1@172.23.122.98',
                                             'ns_1@172.23.122.170'],
                                            ['ns_1@172.23.108.188',
                                             'ns_1@172.23.122.170'],
                                            []}}) exited with reason {unexpected_exit,
                                                                      {'EXIT',
                                                                       <0.6552.22>,
                                                                       {{dcp_wait_for_data_move_failed,
                                                                         "metadata",
                                                                         1023,
                                                                         'ns_1@172.23.122.98',
                                                                         ['ns_1@172.23.108.188',
                                                                          'ns_1@172.23.122.170'],
                                                                         {error,
                                                                          no_stats_for_this_vbucket}},
                                                                        [{ns_single_vbucket_mover,
                                                                          '-wait_dcp_data_move/5-fun-0-',
                                                                          5,
                                                                          [{file,
                                                                            "src/ns_single_vbucket_mover.erl"},
                                                                           {line,
                                                                            451}]},
                                                                         {proc_lib,
                                                                          init_p,3,
                                                                          [{file,
                                                                            "proc_lib.erl"},
                                                                           {line,
                                                                            211}]}]}}}
      

      2023-08-08T09:37:35.615-07:00, ns_orchestrator:0:critical:message(ns_1@172.23.122.170) - Rebalance exited with reason {mover_crashed,
                                    {unexpected_exit,
                                     {'EXIT',<0.6552.22>,
                                      {{dcp_wait_for_data_move_failed,"metadata",
                                        1023,'ns_1@172.23.122.98',
                                        ['ns_1@172.23.108.188',
                                         'ns_1@172.23.122.170'],
                                        {error,no_stats_for_this_vbucket}},
                                       [{ns_single_vbucket_mover,
                                         '-wait_dcp_data_move/5-fun-0-',5,
                                         [{file,"src/ns_single_vbucket_mover.erl"},
                                          {line,451}]},
                                        {proc_lib,init_p,3,
                                         [{file,"proc_lib.erl"},{line,211}]}]}}}}.
      Rebalance Operation Id = 469afa0e65ed41917f3cc9c7050f1586
      

      Note

      Might be a possible dup of MB-58072.
      Same test ran fine on 7.2.1-5878 build.
      There is also an occurence of MB-58159 during the test run.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty