Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38527

[System Test]: Rebalance failed with mover_crashed due to failed_to_update_vbucket_map

    XMLWordPrintable

Details

    • Bug
    • Resolution: User Error
    • Critical
    • 6.5.1
    • 6.5.1
    • ns_server

    Description

      Build: 6.5.1-6288 , also seen on 6.5.1-6284 (but logs got rotated) 

      Test: MH longevity

      Cycle: 2nd

      Day: 1st

      Test Step: Add 1 kv node , failover 2 node (graceful and hard)

      [2020-03-31T16:37:35-07:00, sequoiatools/couchbase-cli:6.5:fb707d] server-add -c 172.23.108.103:8091 --server-add https://172.23.106.100 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      warning using 'json' filter:  unexpected end of JSON input []
      [2020-03-31T16:40:07-07:00, sequoiatools/couchbase-cli:6.5:2013c9] failover -c 172.23.108.103:8091 --server-failover 172.23.104.5:8091 -u Administrator -p password
      [2020-03-31T16:49:32-07:00, sequoiatools/couchbase-cli:6.5:f4f976] failover -c 172.23.108.103:8091 --server-failover 172.23.99.20:8091 -u Administrator -p password --force
      [2020-03-31T16:50:17-07:00, sequoiatools/couchbase-cli:6.5:38655f] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      → 
       
       
      Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
       
       
      docker logs 38655f
      docker start 38655f
       
       
      *Unable to display progress bar on this os
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.
      

      [user:error,2020-03-31T16:51:50.105-07:00,ns_1@172.23.108.103:<0.11245.0>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {mover_crashed,
                                    {unexpected_exit,
                                     {'EXIT',<0.10697.600>,
                                      {failed_to_update_vbucket_map,"STOCK",774,
                                       {error,
                                        [{'ns_1@172.23.120.245',
                                          {exit,
                                           {{nodedown,'ns_1@172.23.120.245'},
                                            {gen_server,call,
                                             [{ns_config_rep,'ns_1@172.23.120.245'},
                                              synchronize_everything,
                                              infinity]}}}}]}}}}}.
      Rebalance Operation Id = 0374bb9768fd351cefe89505d7cd2ea3 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            dfinlay Dave Finlay
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty