Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38045

Rebalance exited with reason {buckets_shutdown_wait_failed,

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.0.0
    • Cheshire-Cat
    • couchbase-bucket
    • Untriaged
    • Unknown

    Description

      On a two node vagrant cluster I had done a graceful failover. When a subsequent delta recovery rebalance was done it failed:

      The ns_server debug.log on node .101 showed

      [user:error,2020-02-24T20:46:35.938Z,ns_1@10.112.210.101:<0.1489.0>:ns_orchestrator:log_rebalance_completion:1446]Rebalance exited with reason {buckets_shutdown_wait_failed,
                                    [{'ns_1@10.112.210.102',
                                      {'EXIT',
                                       {old_buckets_shutdown_wait_failed,
                                        ["beer-sample","default","default2",
                                         "magmaBucket"]}}}]}.
      Rebalance Operation Id = fa34adff8f29885dff361cac9047bbae
      

      The ns_server debug log on node .102 showed

      [ns_server:debug,2020-02-24T20:45:15.936Z,ns_1@10.112.210.102:<0.20587.14>:ns_rebalancer:local_buckets_shutdown_loop:192]Waiting until the following old bucket instances are gone: ["beer-sample",
                                                                  "default",
                                                                  "default2",
                                                                  "magmaBucket"]
      <snip>
      [ns_server:info,2020-02-24T20:46:35.946Z,ns_1@10.112.210.102:rebalance_agent<0.928.0>:rebalance_agent:handle_down:296]Rebalancer process <14531.4056.24> died (reason {buckets_shutdown_wait_failed,
                                                       [{'ns_1@10.112.210.102',
                                                         {'EXIT',
                                                          {old_buckets_shutdown_wait_failed,
                                                           ["beer-sample","default",
                                                            "default2",
                                                            "magmaBucket"]}}}]}).
      

      The memcached.log.000021.txt file on node .102 showed

      2020-02-24T20:44:19.228818+00:00 INFO (magmaBucket) ~VBucket(): vb:1010
      2020-02-24T20:44:19.228855+00:00 INFO (magmaBucket) ~VBucket(): vb:1014
      2020-02-24T20:44:19.228884+00:00 INFO (magmaBucket) ~VBucket(): vb:1018
      2020-02-24T20:44:19.228926+00:00 INFO (magmaBucket) ~VBucket(): vb:1022
      2020-02-24T20:44:24.530928+00:00 WARNING (No Engine) Slow runtime for 'DurabilityTimeoutVisitor on vb:516' on thread nonIO_worker_1: 109 ms
      2020-02-24T20:44:24.572929+00:00 CRITICAL Breakpad caught a crash (Couchbase version 7.0.0-10990). Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/649e0b96-36d1-d37c-0f44925a-6be26824.dmp before terminating.
      2020-02-24T20:44:24.572964+00:00 CRITICAL Stack backtrace of crashed thread:
      2020-02-24T20:44:24.579445+00:00 CRITICAL     /opt/couchbase/bin/memcached() [0x400000+0x13bd7d]
      2020-02-24T20:44:24.579469+00:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3ce) [0x400000+0x152f7e]
      2020-02-24T20:44:24.579479+00:00 CRITICAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x94) [0x400000+0x153294]
      2020-02-24T20:44:24.579486+00:00 CRITICAL     /lib64/libpthread.so.0() [0x7f104857c000+0xf130]
      2020-02-24T20:44:24.579506+00:00 CRITICAL     /lib64/libc.so.6(gsignal+0x39) [0x7f10481bb000+0x355c9]
      2020-02-24T20:44:24.579552+00:00 CRITICAL     /lib64/libc.so.6(abort+0x148) [0x7f10481bb000+0x36cd8]
      2020-02-24T20:44:24.579556+00:00 CRITICAL     /opt/couchbase/bin/../lib/libjemalloc.so.2(je_dallocx+0x72c) [0x7f104a437000+0x202ec]
      2020-02-24T20:44:24.579565+00:00 CRITICAL     /opt/couchbase/bin/memcached(cb_free+0x1a) [0x400000+0x116c4a]
      2020-02-24T20:44:24.579648+00:00 CRITICAL     /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7f1048cb1000+0x8d192]
      2020-02-24T20:44:24.579656+00:00 CRITICAL     /lib64/libpthread.so.0() [0x7f104857c000+0x7bf2]
      2020-02-24T20:44:24.579661+00:00 CRITICAL     /lib64/libpthread.so.0() [0x7f104857c000+0x7e01]
      2020-02-24T20:44:24.579692+00:00 CRITICAL     /lib64/libc.so.6(clone+0x6d) [0x7f10481bb000+0xf61ad]
      

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              steve.watanabe Steve Watanabe
              steve.watanabe Steve Watanabe
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty