Details
-
Bug
-
Resolution: Won't Fix
-
Blocker
-
3.0
-
Security Level: Public
-
None
-
Untriaged
-
Yes
-
June 30 - July 18
Description
- started with a 4 node cluster
- failover + rebalance out 2
- attempted to rebalance out 1 more
- hangs
ns_server is trying to move all vbuckets to node_0 but vb5 isn't being taken over:
vb_0: active
vb_1: active
vb_2: active
vb_3: active
vb_4: active
vb_5: replica
vb_6: active
vb_7: active
according to the upr_stats vb5 takeover flags = 0:
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_flags: 0
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_items_ready: false
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_opaque: 3
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_snap_end_seqno: 10
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_snap_start_seqno: 0
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_start_seqno: 10
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_state: reading
-
-
- I'm not sure why items_ready = false
-
the other streams however were takeover streams:
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_2_flags: 1
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_7_flags: 1
the producer seems to suggest the items were sent already:
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_flags: 0
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_items_ready: false
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_last_sent_seqno: 10
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_memory: 10
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_opaque: 3
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_snap_end_seqno: 0
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_snap_start_seqno: 0
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_start_seqno: 0
eq_uprq:replication:n_1@10.0.0.105->n_0@10.0.0.105:default:stream_5_state: in-memory
also noticed vb5 has an open checkpoint with 0 items if that is of any significance…
vb_5:last_closed_checkpoint_id: 1
vb_5:num_checkpoint_items: 1
vb_5:num_checkpoints: 1
vb_5:num_items_for_persistence: 1
vb_5:num_open_checkpoint_items: 0
vb_5:num_tap_cursors: 0
vb_5:open_checkpoint_id: 2
vb_5:persisted_checkpoint_id: 1
vb_5:state: replica