Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38632

Graceful failover exited with reason badmatch

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • 6.5.1
    • 6.5.1
    • ns_server
    • centos1

    Description

      Build: 6.5.1-6293 

      Test : MH longevity ( -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml )

      Cycle: 2nd

      Day: 1st 

      Test step: Add 1 kv, failover 2 kv (graceful and hard) and rebalance

      [2020-04-06T14:33:12-07:00, sequoiatools/couchbase-cli:6.5:ab132b] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2020-04-06T14:41:00-07:00, sequoiatools/couchbase-cli:6.5:51d47f] failover -c 172.23.108.103:8091 --server-failover 172.23.104.5:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[failover -c 172.23.108.103:8091 --server-failover 172.23.104.5:8091 -u Administrator -p password]
       
       
      docker logs 51d47f
      docker start 51d47f
       
       
      *Unable to display progress bar on this os
      RERROR: Graceful failover failed. See logs for detailed reason. You can try again.
      [2020-04-06T14:51:19-07:00, sequoiatools/couchbase-cli:6.5:2ce681] failover -c 172.23.108.103:8091 --server-failover 172.23.97.239:8091 -u Administrator -p password --force
      [2020-04-06T14:52:26-07:00, sequoiatools/couchbase-cli:6.5:6b1e5a] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      [2020-04-06T15:04:45-07:00, sequoiatools/cmd:295317] 60 

      Graceful failover failed as

      [user:error,2020-04-06T14:43:50.677-07:00,ns_1@172.23.108.103:<0.20123.1>:ns_orchestrator:log_rebalance_completion:1445]Graceful failover exited with reason {{badmatch,
                                             {leader_activities_error,
                                              {default,
                                               {ns_janitor,"NEW_ORDER",
                                                apply_config}},
                                              {quorum_lost,
                                               {lease_lost,'ns_1@172.23.97.121'}}}},
                                            [{ns_janitor,cleanup_apply_config,4,
                                              [{file,"src/ns_janitor.erl"},
                                               {line,213}]},
                                             {ns_janitor,
                                              '-cleanup_membase_bucket/4-fun-0-',4,
                                              [{file,"src/ns_janitor.erl"},
                                               {line,65}]},
                                             {async,'-async_init/4-fun-1-',3,
                                              [{file,"src/async.erl"},{line,197}]}]}.
      Rebalance Operation Id = 7cdf6739997d181fe1884558db4d900c 

      Recent fixes MB-38521 , MB-38487

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            vikas.chaudhary Vikas Chaudhary
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty