Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29113

Rebalance failed due to dcp_consumer_conn moves to idle state

    XMLWordPrintable

Details

    • Bug
    • Resolution: Won't Fix
    • Test Blocker
    • 5.5.0
    • 5.5.0
    • qe
    • Centos 7.4 64-bit
    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Install Couchbase Server 5.5.0-2401 on 2 centos servers.

      Run connection test

      ./testrunner -i 2nodes.ini -t connectionstests.ConnectionTests.create_connections_test,servers_in=1,dgm_run=True

      Test failed during rebalance around 55% to 68% with error : Rebalance stop by janitor.

      Check in debug log in node 224, I found the failure may come from dcp consumer conn

      [ns_server:debug,2018-04-04T11:36:59.361-07:00,ns_1@172.23.121.224:dcp_consumer_conn-default-ns_1@172.23.121.225<0.14982.3>:dcp_proxy:handle_packet:160]Proxy packet: RESPONSE: 0x51 (dcp_add_stream) vbucket = 0 opaque = 0x3FE status = 0x0 (success)
      81 51 00 00
      04 00 00 00
      00 00 00 04
      00 00 03 FE
      00 00 00 00
      00 00 00 00
      00 00 02 07
       
      [rebalance:debug,2018-04-04T11:36:59.361-07:00,ns_1@172.23.121.224:dcp_consumer_conn-default-ns_1@172.23.121.225<0.14982.3>:dcp_consumer_conn:handle_packet:75]Stream has been added for partition 1022, stream opaque = 0x7020000
      [ns_server:debug,2018-04-04T11:36:59.362-07:00,ns_1@172.23.121.224:dcp_consumer_conn-default-ns_1@172.23.121.225<0.14982.3>:dcp_proxy:handle_packet:160]Proxy packet: RESPONSE: 0x51 (dcp_add_stream) vbucket = 0 opaque = 0x3FF status = 0x0 (success)
      81 51 00 00
      04 00 00 00
      00 00 00 04
      00 00 03 FF
      00 00 00 00
      00 00 00 00
      00 00 02 08
       
      [rebalance:debug,2018-04-04T11:36:59.362-07:00,ns_1@172.23.121.224:dcp_consumer_conn-default-ns_1@172.23.121.225<0.14982.3>:dcp_consumer_conn:handle_packet:75]Stream has been added for partition 1023, stream opaque = 0x8020000
      [ns_server:debug,2018-04-04T11:36:59.362-07:00,ns_1@172.23.121.224:dcp_consumer_conn-default-ns_1@172.23.121.225<0.14982.3>:dcp_consumer_conn:maybe_reply_setup_streams:418]Setup stream request completed with ok. Moving to idle state
      [user:info,2018-04-04T11:36:59.362-07:00,ns_1@172.23.121.224:ns_config<0.157.0>:ns_janitor:maybe_stop_rebalance_status:311]Resetting rebalance status since it's not really running
      [ns_server:debug,2018-04-04T11:36:59.419-07:00,ns_1@172.23.121.224:ns_config_log<0.164.0>:ns_config_log:log_common:210]config change:
      {local_changes_count,<<"509d13c921b1afa1ad0184e03f918fed">>} ->
      [{'_vclock',[{<<"509d13c921b1afa1ad0184e03f918fed">>,{673,63690086219}}]}]
      [ns_server:debug,2018-04-04T11:36:59.419-07:00,ns_1@172.23.121.224:ns_config_log<0.164.0>:ns_config_log:log_common:210]config change:
      rebalancer_pid ->
      [{'_vclock',[{<<"509d13c921b1afa1ad0184e03f918fed">>,{2,63690086219}}]}|
       undefined]
      [ns_server:debug,2018-04-04T11:36:59.422-07:00,ns_1@172.23.121.224:ns_config_log<0.164.0>:ns_config_log:log_common:210]config change:
      rebalance_status ->
      [{'_vclock',[{<<"509d13c921b1afa1ad0184e03f918fed">>,{2,63690086219}}]}|
       {none,<<"Rebalance stopped by janitor.">>}]
      [ns_server:debug,2018-04-04T11:36:59.436-07:00,ns_1@172.23.121.224:ns_config_rep<0.14109.3>:ns_config_rep:do_push_keys:330]Replicating some config keys ([rebalance_status,rebalancer_pid,
                                     {local_changes_count,
                                         <<"509d13c921b1afa1ad0184e03f918fed">>}]..)

       

       

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            thuan Thuan Nguyen
            thuan Thuan Nguyen
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty