Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-30811

Healthy cluster node got removed and added back to the cluster

    XMLWordPrintable

Details

    • Untriaged
    • Attaching logs for both the clusters
    • Unknown

    Description

      Cluster member node removed and rebalanced in again.

      Scenario:

      1. Created two 3-node cluster test-couchbase-lkds5 and test-couchbase-lwmmt
      2. Enabled XDCR replication from cluster1 -> cluster2 using couchbase bucket "default"
      3. Rebalancing out cluster2 nodes one by one and replacing it by new node, such that cluster's size will remain the name

      This test was run on Kubernetes environment using Enterprise "Edition 5.5.0 build 2958" docker image couchbase/server:enterprise-5.5.

      enterprise-5.5.0: Pulling from couchbase/server
      Digest: sha256:5228ded10c8fca39e8cea48cd845130d97b3770dc50f336b4214dcb165faaeda
      Status: Image is up to date for couchbase/server:enterprise-5.5.0

      ns_server log file prints:

      [rebalance:info,2018-08-07T05:09:15.706Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:service_rebalancer-index-worker<0.7724.1>:service_rebalancer:rebalance:110]Rebalancing service index with id <<"8410f3574ede98d83e22b0bb9323aac7">>.
      KeepNodes: ['ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc',
                  'ns_1@test-couchbase-lkds5-0001.test-couchbase-lkds5.default.svc']
      EjectNodes: []
      DeltaNodes: []
      [ns_server:info,2018-08-07T05:09:15.763Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:mb_master<0.750.0>:mb_master:master:374]Got master heartbeat from 'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc' when I'm master
      [ns_server:warn,2018-08-07T05:09:15.765Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.3701.0>:leader_lease_acquire_worker:handle_lease_already_acquired:232]Failed to acquire lease from 'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc' because its already taken by {'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc',
                                                                                                                                   <<"466d18dbf6e8f6a64e18be420f2d76a2">>} (valid for 14674ms)
      [stats:warn,2018-08-07T05:09:15.767Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:query_stats_collector<0.440.0>:base_stats_collector:latest_tick:69](Collector: query_stats_collector) Dropped 1 ticks
      [stats:warn,2018-08-07T05:09:15.769Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.4895.0>:base_stats_collector:latest_tick:69](Collector: goxdcr_stats_collector) Dropped 1 ticks
      [stats:warn,2018-08-07T05:09:15.771Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:service_stats_collector-index<0.887.0>:base_stats_collector:latest_tick:69](Collector: service_stats_collector) Dropped 1 ticks
      [stats:warn,2018-08-07T05:09:15.771Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:system_stats_collector<0.427.0>:base_stats_collector:latest_tick:69](Collector: system_stats_collector) Dropped 1 ticks
      [stats:warn,2018-08-07T05:09:15.775Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.445.0>:base_stats_collector:latest_tick:69](Collector: global_stats_collector) Dropped 1 ticks
      [stats:warn,2018-08-07T05:09:15.794Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.4889.0>:base_stats_collector:latest_tick:69](Collector: stats_collector) Dropped 1 ticks
      [user:info,2018-08-07T05:09:15.806Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.7614.1>:ns_rebalancer:orchestrate_failover:94]Failed over ['ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc']: ok
      [ns_server:info,2018-08-07T05:09:15.809Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:leader_quorum_nodes_manager<0.755.0>:leader_quorum_nodes_manager:handle_set_quorum_nodes:136]Updating quorum nodes.
      Old quorum nodes: ['ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc',
                         'ns_1@test-couchbase-lkds5-0001.test-couchbase-lkds5.default.svc',
                         'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc']
      New quorum nodes: ['ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc',
                         'ns_1@test-couchbase-lkds5-0001.test-couchbase-lkds5.default.svc']
      [user:info,2018-08-07T05:09:15.935Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.7614.1>:ns_rebalancer:deactivate_nodes:112]Deactivating failed over nodes ['ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc']
      

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty