Loading...

XML

Word

Printable

Steps to reproduce:

Create a 6 node cluster 3KV-2index-1query.
Create about 10 buckets with 2 indexes on each bucket.
Load the docs until all of the buckets have data greater than memory.
Kill memcached (either by OOM kill or by kill -9) and check that 1 KV node auto fails over.
Do a delta recovery and rebalance.

Doesn't always rollback to zero. I've attached the logs for the time it did. Excerpts from the logs

[root@localhost ~]# cat /opt/couchbase/var/lib/couchbase/logs/indexer.log | grep rollbackAllToZero

2022-07-29T03:02:50.535-07:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM test10

cbcollect

s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.15.zip

s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.17.zip

s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.16.zip

s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.19.zip

s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.100.22.zip

s3://cb-customers-secure/cbse12279rollback3/2022-07-29/collectinfo-2022-07-29t101446-ns_1@172.23.121.215.zip

backports to

MB-53250 [Backport MB-53183 to 7.0.5] Secondary Index rollback to zero after KV node auto failover

is duplicated by

MB-53162 RollbackAll to zero seen in the indexer logs

MB-53263 [System Test] [6.6.5-10108]Disk commit timestamp is not snapshot aligned

MB-53202 Index rollbackAllToZero messages seen with build 6.6.5-10106

[6.6.5 build 10106] - Secondary Index rollback to zero after KV node auto failover