Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-16093

rebalance failed due to dcp_wait_for_data_move_failed,"default",137,

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Major
    • 4.1.0
    • 3.1.0
    • couchbase-bucket
    • Security Level: Public
    • None
    •  centos 6.5 64-bit, 8 core cpu, 30 GB RAM, 80 GB SSD, 10 Git network

    Description

      Install cb server 3.1.0-1776 on 7 nodes
      Create cluster of 7 nodes
      Create defautl bucket with 2 replica
      Load 119 million items with size 100 bytes to default bucket (no view and 100% set) Front load is 250K/second

      Install cb server 3.1.0-1776 on other 5 nodes
      Add new 5 nodes to cluster and remove 5 nodes (swap rebalance)
      Rebalance failed with error memcached disconnect

      Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
      {error,
      timeout}} ns_memcached000 ns_1@10.156.27.78 20:33:33 - Tue Aug 18, 2015
      Bucket "default" loaded on node 'ns_1@10.156.27.78' in 0 seconds. ns_memcached000 ns_1@10.156.27.78 20:26:47 - Tue Aug 18, 2015
      Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
      {error,
      timeout}} ns_memcached000 ns_1@10.156.27.78 20:26:46 - Tue Aug 18, 2015
      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.29088.218>,
      {dcp_wait_for_data_move_failed,"default",137,
      'ns_1@10.11.196.105',
      ['ns_1@10.45.75.76','ns_1@10.156.27.78'],
      {error,no_stats_for_this_vbucket}}}}
      ns_orchestrator002 ns_1@10.156.27.78 20:23:46 - Tue Aug 18, 2015
      <0.28331.218> exited with {unexpected_exit,
      {'EXIT',<0.29088.218>,
      {dcp_wait_for_data_move_failed,"default",
      137,'ns_1@10.11.196.105',
      ['ns_1@10.45.75.76','ns_1@10.156.27.78'],
      {error,no_stats_for_this_vbucket}}}} ns_vbucket_mover000 ns_1@10.156.27.78 20:23:46 - Tue Aug 18, 2015
      Bucket "default" loaded on node 'ns_1@10.156.27.78' in 0 seconds. ns_memcached000 ns_1@10.156.27.78 20:19:04 - Tue Aug 18, 2015
      Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
      {error,
      timeout}} ns_memcached000 ns_1@10.156.27.78 20:19:03 - Tue Aug 18, 2015
      Bucket "default" loaded on node 'ns_1@10.156.27.78' in 0 seconds. ns_memcached000 ns_1@10.156.27.78 20:09:02 - Tue Aug 18, 2015
      Control connection to memcached on 'ns_1@10.156.27.78' disconnected: {badmatch,
      {error,
      timeout}} ns_memcached000 ns_1@10.156.27.78 20:09:02 - Tue Aug 18, 2015
      Bucket "default" loaded on node 'ns_1@10.156.46.169' in 0 seconds. ns_memcached000 ns_1@10.156.46.169 19:16:11 - Tue Aug 18, 2015
      Deleting old data files of bucket "default" ns_storage_conf000 ns_1@10.156.46.169 19:16:10 - Tue Aug 18, 2015
      Bucket "default" loaded on node 'ns_1@10.229.83.110' in 0 seconds. ns_memcached000 ns_1@10.229.83.110 19:16:09 - Tue Aug 18, 2015
      Deleting old data files of bucket "default" ns_storage_conf000 ns_1@10.229.83.110 19:16:08 - Tue Aug 18, 2015
      Bucket "default" rebalance appears to be swap rebalance ns_vbucket_mover000 ns_1@10.156.27.78 19:16:02 - Tue Aug 18, 2015
      Bucket "default" loaded on node 'ns_1@10.45.75.76' in 0 seconds. ns_memcached000 ns_1@10.45.75.76 19:16:01 - Tue Aug 18, 2015
      Deleting old data files of bucket "default" ns_storage_conf000 ns_1@10.45.75.76 19:16:01 - Tue Aug 18, 2015
      Started rebalancing bucket default ns_rebalancer000 ns_1@10.156.27.78 19:15:58 - Tue Aug 18, 2015
      Starting rebalance, KeepNodes = ['ns_1@10.11.196.105','ns_1@10.156.27.78',
      'ns_1@10.45.75.76','ns_1@10.141.207.54',
      'ns_1@10.154.33.77','ns_1@10.156.46.169',
      'ns_1@10.229.83.110'], EjectNodes = ['ns_1@10.157.170.23',
      'ns_1@10.152.47.238',
      'ns_1@10.182.90.53',
      'ns_1@10.141.15.231',
      'ns_1@10.182.89.250'], Failed over and being ejected nodes = []; no delta recovery nodes
      ns_orchestrator004 ns_1@10.156.27.78 19:15:57 - Tue Aug 18, 2015

      I will try rebalance again and update this ticket

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              manu Manu Dhundi (Inactive)
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty