Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-41277

[System Test]: Graceful failover exited with badmatch , leader_activities_error

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • 6.6.0
    • 6.6.0
    • ns_server

    Description

      Build: 6.6.0-7918 (MP1)

      Test: MH longevity

      Cycle:3

      Day: 2nd

      Test Step

      [2020-09-02T18:57:34-07:00, sequoiatools/couchbase-cli:6.5:dd7d72] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      [2020-09-02T19:04:29-07:00, sequoiatools/couchbase-cli:6.5:34b52f] failover -c 172.23.108.103:8091 --server-failover 172.23.97.239:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[failover -c 172.23.108.103:8091 --server-failover 172.23.97.239:8091 -u Administrator -p password]
       
       
      docker logs 34b52f
      docker start 34b52f
       
       
      sWARNING: couchbase-cli version 6.5.0-3216-enterprise does not match couchbase server version 6.6.0-7918-enterprise
      *Unable to display progress bar on this os
      RERROR: Graceful failover failed. See logs for detailed reason. You can try again.
      [2020-09-02T19:15:24-07:00, sequoiatools/couchbase-cli:6.5:f074ca] failover -c 172.23.108.103:8091 --server-failover 172.23.99.20:8091 -u Administrator -p password --force
      [2020-09-02T19:16:21-07:00, sequoiatools/couchbase-cli:6.5:2e2e36] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      [2020-09-02T19:29:48-07:00, sequoiatools/cmd:8be21f] 60 

      Error logs

      [ns_server:error,2020-09-02T19:09:06.631-07:00,ns_1@172.23.108.103:<0.32085.869>:leader_activities:report_error:1011]Activity {default,{ns_janitor,"HISTORY",apply_config}} failed with error {quorum_lost,
                                                                                {lease_lost,
                                                                                 'ns_1@172.23.104.67'}}
      [user:error,2020-09-02T19:09:06.633-07:00,ns_1@172.23.108.103:<0.11364.0>:ns_orchestrator:log_rebalance_completion:1445]Graceful failover exited with reason {{badmatch,
                                             {leader_activities_error,
                                              {default,
                                               {ns_janitor,"HISTORY",apply_config}},
                                              {quorum_lost,
                                               {lease_lost,'ns_1@172.23.104.67'}}}},
                                            [{ns_janitor,cleanup_apply_config,4,
                                              [{file,"src/ns_janitor.erl"},
                                               {line,213}]},
                                             {ns_janitor,
                                              '-cleanup_membase_bucket/4-fun-0-',4,
                                              [{file,"src/ns_janitor.erl"},
                                               {line,65}]},
                                             {async,'-async_init/4-fun-1-',3,
                                              [{file,"src/async.erl"},{line,197}]}]}.
      Rebalance Operation Id = e30cc2fa709686920d4b76d18a97dbec 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty