Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59833

Rebalance exited with reason {mover_crashed,{unexpected_exit,{'EXIT',<0.24040.2>,{{wait_seqno_persisted_failed,"bucket-1"

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Test that led to failure:
      1. Created a 3 node cluster.

      2. Created a bucket and loaded close to 3M docs.

      3. Started a 'rebalance in' operation by adding a new node.

      4. While the rebalance was in progress called the API for password rotation(/node/controller/rotateInternalCredentials). Password rotation completed even though it was expected to fail with the following error

      ["System is being reconfigured. Please try later."]  

      5.Rebalance exited with the following error

      2023-11-26T20:38:56.510-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.121.21) - Rebalance exited with reason {mover_crashed,                              {unexpected_exit,                               {'EXIT',<0.24040.2>,                                {{wait_seqno_persisted_failed,"bucket-1",320,                                  2862,                                  [{'ns_1@172.23.121.199',                                    {'EXIT',                                     {shutdown,                                      {gen_server,call,                                       [{'janitor_agent-bucket-1',                                         'ns_1@172.23.121.199'},                                        {if_rebalance,<0.25080.1>,                                         {wait_seqno_persisted,320,2862}},                                        infinity]}}}}]},                                 [{ns_single_vbucket_mover,                                   '-wait_seqno_persisted_many/5-fun-2-',5,                                   [{file,"src/ns_single_vbucket_mover.erl"},                                    {line,476}]},                                  {proc_lib,init_p,3,                                   [{file,"proc_lib.erl"},{line,225}]}]}}}}. 

      TAF script to reproduce:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-security-vset00-00-security-password-rotation/testexec.101257.ini get-cbcollect-info=True,upgrade_version=7.6.0-1845,sirius_url=http://172.23.120.103:4000 -t security.internal_password.InternalUserPassword.test_password_rotation_rebalance,nodes_init=3,rebalance_type=rebalance_in,nodes_in=1' 

      TAF ref job: debian-security_security-password-rotation

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mohsin.ahmed Mohsin Ahmed
            mohsin.ahmed Mohsin Ahmed
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty