Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19424

gracefull failover: Rebalance exited with reason {buckets_shutdown_wait_failed

    XMLWordPrintable

Details

    • Untriaged
    • Ubuntu 64-bit
    • No

    Description

      http://perf.jenkins.couchbase.com/job/ares/3499/console

      Rebalance exited with reason {buckets_shutdown_wait_failed,
                                    [{'ns_1@172.23.96.14',
                                      {'EXIT',
                                       {old_buckets_shutdown_wait_failed,
                                        ["bucket-1"]}}}]}
      ns_orchestrator 002	ns_1@172.23.96.11	7:17:19 AM Thu Apr 28, 2016
      Failed to wait deletion of some buckets on some nodes: [{'ns_1@172.23.96.14',
                                                               {'EXIT',
                                                                {old_buckets_shutdown_wait_failed,
                                                                 ["bucket-1"]}}}]
      ns_rebalancer 000	ns_1@172.23.96.11	7:17:19 AM Thu Apr 28, 2016
      Starting rebalance, KeepNodes = ['ns_1@172.23.96.11','ns_1@172.23.96.12',
                                       'ns_1@172.23.96.13','ns_1@172.23.96.14'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator 004	ns_1@172.23.96.11	7:16:59 AM Thu Apr 28, 2016
      Shutting down bucket "bucket-1" on 'ns_1@172.23.96.14' for deletion	ns_memcached 000	ns_1@172.23.96.14	7:16:58 AM Thu Apr 28, 2016
      Failed over 'ns_1@172.23.96.14': ok	ns_rebalancer 000	ns_1@172.23.96.11	7:16:57 AM Thu Apr 28, 2016
      Starting failing over 'ns_1@172.23.96.14'	ns_rebalancer 000	ns_1@172.23.96.11	7:16:57 AM Thu Apr 28, 2016
      Bucket "bucket-1" rebalance does not seem to be swap rebalance	ns_vbucket_mover 000	ns_1@172.23.96.11	7:16:45 AM Thu Apr 28, 2016
      Starting vbucket moves for graceful failover of 'ns_1@172.23.96.14'	ns_rebalancer 000	ns_1@172.23.96.11	7:16:45 AM Thu Apr 28, 2016
      User-triggered compaction of bucket `bucket-1` completed.	compaction_new_daemon 000	ns_1@172.23.96.14	6:53:31 AM Thu Apr 28, 2016
      User-triggered compaction of bucket `bucket-1` completed.	compaction_new_daemon 000	ns_1@172.23.96.13	6:53:28 AM Thu Apr 28, 2016
      User-triggered compaction of bucket `bucket-1` completed.	compaction_new_daemon 000	ns_1@172.23.96.12	6:53:06 AM Thu Apr 28, 2016
      User-triggered compaction of bucket `bucket-1` completed.	compaction_new_daemon 000	ns_1@172.23.96.11	6:52:59 AM Thu Apr 28, 2016
      Enabled auto-failover with timeout 120	auto_failover 000	ns_1@172.23.96.11	5:52:35 AM Thu Apr 28, 2016

      http://perf.jenkins.couchbase.com/job/ares/3496/artifact/*zip*/archive.zip

      Attachments

        1. memory-usage-4.1.1.png
          memory-usage-4.1.1.png
          296 kB
        2. memory-usage-watson.png
          memory-usage-watson.png
          299 kB
        3. Resident-ratios-4.1.1.png
          Resident-ratios-4.1.1.png
          217 kB
        4. Resident-ratios-watson.png
          Resident-ratios-watson.png
          212 kB
        For Gerrit Dashboard: MB-19424
        # Subject Branch Project Status CR V

        Activity

          People

            owend Daniel Owen
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty