Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7968

rebalance failed with reason bulk_set_vbucket_state_failed linked_process_died

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.0
    • Component/s: None
    • Security Level: Public
    • Labels:
      None
    • Environment:

      Description

      test to reproduce -t viewtests.ViewFailoverTests.test_view_failover_multiple_design_docs_x_node_replica_y,num-design-docs=20,num-docs=10000,replica=2,failover-factor=2
      1 bucket/ 5 nodes cluster/ 10K items/ 2 replica/ 20 ddocs
      2 nodes was failed over and rebalance started

      Rebalance exited with reason {{bulk_set_vbucket_state_failed,
      [{'ns_1@10.3.3.28',
      {'EXIT',
      {{

      {linked_process_died,<12880.30990.118>, normal}

      ,
      {gen_server,call,
      ['capi_set_view_manager-default'

      {set_vbucket_states, [active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,missing,\n missing,missing,missing,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,active,active,\n active,active,active,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica,replica,replica,replica,\n replica],\n [undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined,undefined,undefined,\n undefined]}

      ,\n infinity]}},\n {gen_server,call,\n [

      {'janitor_agent-default',\n 'ns_1@10.3.3.28'}

      ,\n {if_rebalance,<0.25656.20>,\n {update_vbucket_state,114,replica,\n undefined,'ns_1@10.3.3.34'}},\n infinity]}}}}]},\n [

      {janitor_agent,bulk_set_vbucket_state,4}

      ,\n

      {ns_vbucket_mover,\n update_replication_post_move,3}

      ,\n

      {ns_vbucket_mover,on_move_done,2}

      ,\n

      {gen_server,handle_msg,5}

      ,\n

      {proc_lib,init_p_do_apply,3}

      ]}

      attaching logs

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Show
        iryna iryna added a comment - https://s3.amazonaws.com/bugdb/jira/MB-7968/16fc64ab/10.3.3.28-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7968/16fc64ab/10.3.3.34-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7968/16fc64ab/10.3.3.35-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7968/16fc64ab/10.3.3.36-diag.txt.gz https://s3.amazonaws.com/bugdb/jira/MB-7968/16fc64ab/10.3.3.37-diag.txt.gz
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Caused by timeouts. I need to ensure your environment is ok.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Caused by timeouts. I need to ensure your environment is ok.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        iryna, can u send output of measure-sched-delays? thanks.

        Show
        maria Maria McDuff (Inactive) added a comment - iryna, can u send output of measure-sched-delays? thanks.
        Hide
        iryna iryna added a comment -

        not reproduced

        Show
        iryna iryna added a comment - not reproduced

          People

          • Assignee:
            iryna iryna
            Reporter:
            iryna iryna
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes