Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50006

[System Test] Indexer crashed during rebalance - "panic: Timestamp mismatch between snapshot"

    XMLWordPrintable

Details

    Description

      Build : 7.1.0-1861
      Test : -test tests/integration/neo/test_neo_couchstore_milestone3.yml -scope tests/integration/neo/scope_couchstore.yml
      Scale : 3
      Iteration : 1st

      While rebalancing out 2 failed over KV nodes, indexer on 172.23.104.70 crashed.

      2021-12-08T12:50:49.223-08:00 [Warn] TransferToken43:ec:38:32:e6:8a:a7:fa Detected Invalid State Change Notification. Token Id Rebalancer::checkValidNotifyStateDest: Local State TransferTokenMerge Metakv State TransferTokenMerge
      2021-12-08T12:50:49.223-08:00 [Info] Indexer::handleMergePartition Source 11054455547261089813 Target 3991167096929591284
      2021-12-08T12:50:49.223-08:00 [Info] MergePartitions: keyspaceId bucket7 streamId MAINT_STREAM
      2021-12-08T12:50:49.223-08:00 [Info] MergePartition: Merge instance 11054455547261089813 to instance 3991167096929591284
      2021-12-08T12:50:49.223-08:00 [Warn] KeyPartitionContainer: Invalid Partition Id 4
      2021-12-08T12:50:49.229-08:00 [Info] Rebalancer::decodeTransferToken TransferToken TransferToken43:ec:38:32:e6:8a:a7:fa  MasterId: ff1673bbe6c3b522825cdfdbd4b413f4 SourceId: af4a76acbe24f9aeaad58613caf94c0d (172.23.121.117:8091) DestId: 9a886614b272fe4ca776d0e73b414b62 (172.23.104.70:8091) RebalId: 5836c835945da68fa1e100c0f6231594 State: TransferTokenInProgress BuildSource: Dcp TransferMode: Move InstId: 11054455547261089813 RealInstId: 3991167096929591284 Partitions: [4] Versions: [4] Inst:
              InstId: 3991167096929591284
              Defn: DefnId: 8157248317897210410 Name: idx9_wTuUtZ Using: plasma Bucket: bucket7 Scope/Id: scope_3/b Collection/Id: coll_1/18 IsPrimary: false NumReplica: 2 InstVersion: 4
                      SecExprs: <ud>([(all (array flatten_keys((`r`.`author`), ((`r`.`ratings`).`Rooms`)) for `r` in `reviews` end)) `free_parking`])</ud>
                      Desc: [false false false]
                      PartitionScheme: KEY
                      HashScheme: CRC32 PartitionKeys: [(meta().`id`)] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
              State: INDEX_STATE_ACTIVE
              RState: RebalActive
              Stream: NIL_STREAM
              Version: 3
              ReplicaId: 2
              PartitionContainer: <nil>
      2021-12-08T12:50:49.230-08:00 [Warn] TransferToken43:ec:38:32:e6:8a:a7:fa Detected Invalid State Change Notification. Token Id Rebalancer::checkValidNotifyStateDest: Local State TransferTokenMerge Metakv State TransferTokenInProgress
      panic: Timestamp mismatch between snapshot
       target bucket: bucket7, scopeId: , collectionId: , vbuckets: 317 Crc64: 13121732074923241505 snapType INMEM_SNAP -
          {vbno, vbuuid, manifest, seqno, snapshot-start, snapshot-end}
          {    0     c10dc885e7fc 24      59526          0      59526}
          {    1     5ba67b4a3744 24      46427          0      46427}
          {    2     bf898c7613eb 24      75341          0      75341}
          {    3     1fd1d4322faf 24      63692          0      63692}
      ...
      ...
      ...
          { 1020     6ea2902f37b2 24      67849          0      67849}
          { 1021     206d533407b7 24      63475          0      63475}
          { 1022     7e394a761f80 24      66284          0      66284}
          { 1023     6a28aee31079 24      64119          0      64119}
       
       
       
      goroutine 36742359 [running]:
      github.com/couchbase/indexing/secondary/common.CrashOnError(...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:464
      github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED.func1(0xc00f5e91e0, 0xc013afcad4, 0x24, 0xc013eda620, 0x20, 0xc013eda640, 0x20, 0xc013eda660, 0x20, 0xc013eda680, ...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2234 +0x657
      created by github.com/couchbase/indexing/secondary/indexer.(*Rebalancer).destTokenToMergeOrReadyLOCKED
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalancer.go:2204 +0xf8
      Initializing write barrier = 8000
      2021-12-08T12:50:50.017-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem -ipv4=required -ipv6=off -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/data/couchbase/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=9a886614b272fe4ca776d0e73b414b62 -isEnterprise=true]
      

      Attachments

        Issue Links

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty