Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-25481

4.6.3 System Test : Rebalance failure when taking out failed over nodes

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 4.6.3
    • 4.6.3
    • view-engine
    • Untriaged
    • Unknown

    Description

      Build : 4.6.3-4084

      The following step in the integration system test failed on the 4th day. It had succeeded previously.

      [2017-07-29T19:54:25-04:00, sequoiatools/couchbase-cli:5f05a9] server-add -c 172.23.106.14:8091 --server-add 172.23.106.213:8091 -u Administrator -p password --server-add-username Administrator --server-add-password password
      [2017-07-29T19:54:38-04:00, sequoiatools/couchbase-cli:1a3f1f] failover -c 172.23.106.14:8091 --server-failover 172.23.105.83:8091 -u Administrator -p password
      [2017-07-29T20:02:57-04:00, sequoiatools/couchbase-cli:1b6379] failover -c 172.23.106.14:8091 --server-failover 172.23.105.63:8091 -u Administrator -p password --force
      [2017-07-29T20:03:09-04:00, sequoiatools/couchbase-cli:5ad0df] rebalance -c 172.23.106.14:8091 -u Administrator -p password

      Error occurred on container, try:
      docker logs 5ad0df
      docker start 5ad0df

      The following error can be seen in the info.log for the same activity :
      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.26743.1475>,
      {{error,
      {badrpc,
      {'EXIT',
      {{error,
      <<"Partition 257 not in active nor passive set">>},
      {gen_server,call,
      ['capi_set_view_manager-default',

      {wait_index_updated,257}

      ,
      infinity]}}}}},
      {gen_server,call,
      [

      {'janitor_agent-default', 'ns_1@172.23.106.213'}

      ,
      {if_rebalance,<0.8465.1474>,
      {dcp_takeover,'ns_1@172.23.99.168',376}},
      infinity]}}}}

      Test console : http://qa.sc.couchbase.com/job/ubuntu-systest-launcher/372/console

      Logs :
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.105.60.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.105.61.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.105.62.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.105.63.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.105.83.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.106.14.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.106.213.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.106.96.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.99.168.zip
      https://s3.amazonaws.com/bugdb/MB-25481/collectinfo-2017-07-30T050352-ns_1%40172.23.99.253.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              jeelan.poola Jeelan Poola
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty