[System Test] Indexer crashed during rebalance - "panic: Timestamp mismatch between snapshot"

2021-12-08T12:50:49.223-08:00 [Warn] TransferToken43:ec:38:32:e6:8a:a7:fa Detected Invalid State Change Notification. Token Id Rebalancer::checkValidNotifyStateDest: Local State TransferTokenMerge Metakv State TransferTokenMerge

2021-12-08T12:50:49.223-08:00 [Info] Indexer::handleMergePartition Source 11054455547261089813 Target 3991167096929591284

2021-12-08T12:50:49.223-08:00 [Info] MergePartitions: keyspaceId bucket7 streamId MAINT_STREAM

2021-12-08T12:50:49.223-08:00 [Info] MergePartition: Merge instance 11054455547261089813 to instance 3991167096929591284

2021-12-08T12:50:49.223-08:00 [Warn] KeyPartitionContainer: Invalid Partition Id 4

2021-12-08T12:50:49.229-08:00 [Info] Rebalancer::decodeTransferToken TransferToken TransferToken43:ec:38:32:e6:8a:a7:fa  MasterId: ff1673bbe6c3b522825cdfdbd4b413f4 SourceId: af4a76acbe24f9aeaad58613caf94c0d (172.23.121.117:8091) DestId: 9a886614b272fe4ca776d0e73b414b62 (172.23.104.70:8091) RebalId: 5836c835945da68fa1e100c0f6231594 State: TransferTokenInProgress BuildSource: Dcp TransferMode: Move InstId: 11054455547261089813 RealInstId: 3991167096929591284 Partitions: [4] Versions: [4] Inst:

        InstId: 3991167096929591284

        Defn: DefnId: 8157248317897210410 Name: idx9_wTuUtZ Using: plasma Bucket: bucket7 Scope/Id: scope_3/b Collection/Id: coll_1/18 IsPrimary: false NumReplica: 2 InstVersion: 4

                SecExprs: <ud>([(all (array flatten_keys((`r`.`author`), ((`r`.`ratings`).`Rooms`)) for `r` in `reviews` end)) `free_parking`])</ud>

                Desc: [false false false]

                PartitionScheme: KEY

                HashScheme: CRC32 PartitionKeys: [(meta().`id`)] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false

        State: INDEX_STATE_ACTIVE

        RState: RebalActive

        Stream: NIL_STREAM

        Version: 3

        ReplicaId: 2

        PartitionContainer: <nil>

2021-12-08T12:50:49.230-08:00 [Warn] TransferToken43:ec:38:32:e6:8a:a7:fa Detected Invalid State Change Notification. Token Id Rebalancer::checkValidNotifyStateDest: Local State TransferTokenMerge Metakv State TransferTokenInProgress

panic: Timestamp mismatch between snapshot

 target bucket: bucket7, scopeId: , collectionId: , vbuckets: 317 Crc64: 13121732074923241505 snapType INMEM_SNAP -

    {vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end}

    {    0     c10dc885e7fc 24      59526          0      59526}

    {    1     5ba67b4a3744 24      46427          0      46427}

    {    2     bf898c7613eb 24      75341          0      75341}

    {    3     1fd1d4322faf 24      63692          0      63692}

...

...

...

    { 1020     6ea2902f37b2 24      67849          0      67849}

    { 1021     206d533407b7 24      63475          0      63475}

    { 1022     7e394a761f80 24      66284          0      66284}

    { 1023     6a28aee31079 24      64119          0      64119}

goroutine 36742359 [running]:

github.com/couchbase/indexing/secondary/common.CrashOnError(...)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:464

github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED.func1(0xc00f5e91e0, 0xc013afcad4, 0x24, 0xc013eda620, 0x20, 0xc013eda640, 0x20, 0xc013eda660, 0x20, 0xc013eda680, ...)

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2234 +0x657

created by github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED

        /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2204 +0xf8

Initializing write barrier = 8000

2021-12-08T12:50:50.017-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem -ipv4=required -ipv6=off -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/data/couchbase/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=9a886614b272fe4ca776d0e73b414b62 -isEnterprise=true]