Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
5.5.0
-
Untriaged
-
Attaching logs for both the clusters
-
Unknown
Description
Cluster member node removed and rebalanced in again.
Scenario:
- Created two 3-node cluster test-couchbase-lkds5 and test-couchbase-lwmmt
- Enabled XDCR replication from cluster1 -> cluster2 using couchbase bucket "default"
- Rebalancing out cluster2 nodes one by one and replacing it by new node, such that cluster's size will remain the name
This test was run on Kubernetes environment using Enterprise "Edition 5.5.0 build 2958" docker image couchbase/server:enterprise-5.5.
enterprise-5.5.0: Pulling from couchbase/server
|
Digest: sha256:5228ded10c8fca39e8cea48cd845130d97b3770dc50f336b4214dcb165faaeda
|
Status: Image is up to date for couchbase/server:enterprise-5.5.0
|
ns_server log file prints:
[rebalance:info,2018-08-07T05:09:15.706Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:service_rebalancer-index-worker<0.7724.1>:service_rebalancer:rebalance:110]Rebalancing service index with id <<"8410f3574ede98d83e22b0bb9323aac7">>.
|
KeepNodes: ['ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc',
|
'ns_1@test-couchbase-lkds5-0001.test-couchbase-lkds5.default.svc']
|
EjectNodes: []
|
DeltaNodes: []
|
[ns_server:info,2018-08-07T05:09:15.763Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:mb_master<0.750.0>:mb_master:master:374]Got master heartbeat from 'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc' when I'm master
|
[ns_server:warn,2018-08-07T05:09:15.765Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.3701.0>:leader_lease_acquire_worker:handle_lease_already_acquired:232]Failed to acquire lease from 'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc' because its already taken by {'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc',
|
<<"466d18dbf6e8f6a64e18be420f2d76a2">>} (valid for 14674ms)
|
[stats:warn,2018-08-07T05:09:15.767Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:query_stats_collector<0.440.0>:base_stats_collector:latest_tick:69](Collector: query_stats_collector) Dropped 1 ticks
|
[stats:warn,2018-08-07T05:09:15.769Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.4895.0>:base_stats_collector:latest_tick:69](Collector: goxdcr_stats_collector) Dropped 1 ticks
|
[stats:warn,2018-08-07T05:09:15.771Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:service_stats_collector-index<0.887.0>:base_stats_collector:latest_tick:69](Collector: service_stats_collector) Dropped 1 ticks
|
[stats:warn,2018-08-07T05:09:15.771Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:system_stats_collector<0.427.0>:base_stats_collector:latest_tick:69](Collector: system_stats_collector) Dropped 1 ticks
|
[stats:warn,2018-08-07T05:09:15.775Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.445.0>:base_stats_collector:latest_tick:69](Collector: global_stats_collector) Dropped 1 ticks
|
[stats:warn,2018-08-07T05:09:15.794Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.4889.0>:base_stats_collector:latest_tick:69](Collector: stats_collector) Dropped 1 ticks
|
[user:info,2018-08-07T05:09:15.806Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.7614.1>:ns_rebalancer:orchestrate_failover:94]Failed over ['ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc']: ok
|
[ns_server:info,2018-08-07T05:09:15.809Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:leader_quorum_nodes_manager<0.755.0>:leader_quorum_nodes_manager:handle_set_quorum_nodes:136]Updating quorum nodes.
|
Old quorum nodes: ['ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc',
|
'ns_1@test-couchbase-lkds5-0001.test-couchbase-lkds5.default.svc',
|
'ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc']
|
New quorum nodes: ['ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc',
|
'ns_1@test-couchbase-lkds5-0001.test-couchbase-lkds5.default.svc']
|
[user:info,2018-08-07T05:09:15.935Z,ns_1@test-couchbase-lkds5-0000.test-couchbase-lkds5.default.svc:<0.7614.1>:ns_rebalancer:deactivate_nodes:112]Deactivating failed over nodes ['ns_1@test-couchbase-lkds5-0002.test-couchbase-lkds5.default.svc']
|
Attachments
Issue Links
- blocks
-
K8S-459 Stabilize all regression for new features
- Closed