{DCP}:: Highest Seq Number changes in Vbuckets after Failover+Rebalance

1159, vbuckets=1024

Test Case:: ./testrunner -i ../palm.ini -t failover.failovertests.FailoverTests.test_failover_normal,replica=2,load_ratio=1,skip_cleanup=True,num_failed_nodes=1

1. Create 7 Node cluster
2. Create default bucket with 1000 Items with replica=2
3. Hard Failover 1 Node (non-master) '10.6.2.146' and then rebalance
4. Drain all queues and make sure replication is complete

After Step 4 we checked for high seq number as compared to before failover and rebalance

Our condition for check fails

vb_371 :

{'high_seqno': 'Condition Fail:: 3 == 0'}

vb_370 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_373 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_426 :

{'high_seqno': 'Condition Fail:: 3 == 0'}

vb_375 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_377 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_196 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_379 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_511 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_193 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_392 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_406 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_405 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_404 :

{'high_seqno': 'Condition Fail:: 3 == 0'}

vb_403 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_402 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_400 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_646 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_645 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_409 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_408 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_501 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_934 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_938 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_429 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_428 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_398 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_506 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_424 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_504 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_505 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_502 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_420 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_500 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_390 :

{'high_seqno': 'Condition Fail:: 3 == 0'}

vb_28 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_29 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_372 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_932 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_412 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_191 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_394 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_950 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_410 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_641 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_948 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_32 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_46 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_47 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_643 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_488 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_391 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_647 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_634 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_416 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_385 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_418 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_381 :

{'high_seqno': 'Condition Fail:: 3 == 0'}

vb_384 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_386 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_438 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_388 :

{'high_seqno': 'Condition Fail:: 3 == 0'}

vb_640 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_434 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_435 :

{'high_seqno': 'Condition Fail:: 1 == 0'}

vb_432 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_510 :

{'high_seqno': 'Condition Fail:: 1 == 3'}

vb_430 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_943 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_490 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_491 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_492 :

{'high_seqno': 'Condition Fail:: 0 == 3'}

vb_493 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_494 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_496 :

{'high_seqno': 'Condition Fail:: 1 == 2'}

vb_933 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_946 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_944 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

vb_945 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_942 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_427 :

{'high_seqno': 'Condition Fail:: 2 == 0'}

vb_940 :

{'high_seqno': 'Condition Fail:: 0 == 1'}

vb_30 :

{'high_seqno': 'Condition Fail:: 0 == 2'}

This test case was passing in passing in 1157 build, failed for 1160 as well

We compare active vs replica highest seq no. and then do comparison between current and previous numbers per vbucket.

This test is a part of our sanity as well

We did data analysis, there was no difference. Only the highest seq no changed

Will attach logs and data files

This test is repro with cluster run as well