Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-26107

Rebalance failed due to dcp_wait_for_data_move_failed error on centos longevity cluster

    XMLWordPrintable

Details

    Description

      Build : 5.0.0-3519

      On the CentOS longevity run with 5.0.0-3519 (RC6), which is in the 3rd day of the longevity, one rebalance failed due to the following error. This rebalance was following failover of 2 data nodes. The subsequent rebalance was successful.

       

      Rebalance exited with reason {unexpected_exit, {'EXIT',<0.4904.826>, {dcp_wait_for_data_move_failed,"default",1021, 'ns_1@172.23.108.103', ['ns_1@172.23.108.104'], {error,no_stats_for_this_vbucket}}}}

      <0.8095.826> exited with {unexpected_exit, {'EXIT',<0.4904.826>, {dcp_wait_for_data_move_failed,"default", 1021,'ns_1@172.23.108.103', ['ns_1@172.23.108.104'], {error,no_stats_for_this_vbucket}}}}

      Sequoia console log

       

      [2017-09-19T01:10:21-07:00, sequoiatools/couchbase-cli:2afe77] server-add -c 172.23.108.103:8091 --server-add 172.23.108.104:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data
      		 
      [2017-09-19T01:11:59-07:00, sequoiatools/couchbase-cli:f9c4f3] failover -c 172.23.108.103:8091 --server-failover 172.23.99.20:8091 -u Administrator -p password
      		 
      [2017-09-19T02:17:06-07:00, sequoiatools/couchbase-cli:9e7b57] failover -c 172.23.108.103:8091 --server-failover 172.23.99.21:8091 -u Administrator -p password --force
      		 
      [2017-09-19T02:17:23-07:00, sequoiatools/couchbase-cli:bea8cf] rebalance -c 172.23.108.103:8091 -u Administrator -p password
      		 
      →
      		 
      Error occurred on container - b9b5b4a:[rebalance -c 172.23.108.103:8091 -u Administrator -p password]
      		 
      docker logs bea8cf
      		 
      docker start bea8cf 
      		 
      *Unable to display progress bar on this os 
      		 
      JERROR: Rebalance failed. See logs for detailed reason. You can try again.

       

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-25822 Dcp Noop disconnects due to not receiving a response within sufficient time

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-26149 rebalance exited with dcp_wait_for_data_move_failed - centos longevity

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-29965 rebalance failure in ubuntu longevity with message {error,no_stats_for_this_vbucket}

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty