Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-10830

XDCR+UPR : Rebalance during replication fails with "Unexpected response. Unrecognised opaque ~p (~p, ~p)"

    XMLWordPrintable

Details

    • Untriaged
    • No

    Description

      Scenario
      --------------
      -Uni xdcr between 2 one node clusters.
      -Rebalance in one node on destination during replication between source -> destination.
      -Rebalance fails with the following error

      Rebalance exited with reason {unexpected_exit,
      {'EXIT',<0.18186.1>,
      {wait_seqno_persisted_failed,"default",975,
      4807,
      [{'ns_1@10.3.4.189',
      {'EXIT',
      {{{{{badmatch,
      [{<18638.6529.0>,
      {done,exit,
      {{badarg,
      [{io_lib,format,
      ["Unexpected response. Unrecognised opaque ~p (~p, ~p)",
      [978,

      {mc_header,81,0,0,0,4,4, 978}

      ]]},

      {'ale_logger-rebalance',warn,5}

      ,

      {upr_consumer_conn, process_add_close_stream_response, 3}

      ,

      {upr_consumer_conn, handle_packet,5}

      ,

      {upr_proxy,handle_packet,4}

      ,

      {mc_replication,process_data,3}

      ,

      {mc_replication,process_data,4}

      ,

      {upr_proxy,handle_info,2}

      ]},
      {gen_server,call,
      [<18638.5412.0>,

      {setup_streams, [973,974,975,976]}

      ,
      infinity]}},
      [

      {gen_server,call,3}

      ,

      {upr_replicator, '-spawn_and_wait/1-fun-0-', 1}

      ]}}]},
      [

      {misc, sync_shutdown_many_i_am_trapping_exits, 1}

      ,

      {upr_replicator,spawn_and_wait,1}

      ,

      {upr_replicator,handle_call,3}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]},
      {gen_server,call,
      ['upr_replicator-default-ns_1@10.3.4.188',

      {setup_replication,[973,974,975,976]}

      ,
      infinity]}},
      {gen_server,call,
      ['replication_manager-default',

      {change_vbucket_replication,978, undefined}

      ,
      infinity]}},
      {gen_server,call,
      [

      {'janitor_agent-default', 'ns_1@10.3.4.189'}

      ,
      {if_rebalance,<0.16552.1>,
      {wait_seqno_persisted,975,4807}},
      infinity]}}}}]}}}

      Observation
      ------------------

      • Tried again during replication from source -> dest. Rebalance failed for the second time..
      • Paused replication at source, rebalance proceeds to completion.

      Cluster setup
      ---------------------
      Source : 10.3.4.186
      Dest : 10.3.4.188 <-- rebalance in 10.3.4.189

      Adding cbcollect info.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            alkondratenko Aleksey Kondratenko (Inactive)
            apiravi Aruna Piravi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty