Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60759

[System Test] Panic seen - inconsistent internal state

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      Seen on both 108 and 109 -

      On 109

      2024-02-11T08:01:56.665-08:00 [Error] Indexer::handleUpdateIndexRState Unable to find Index 2162545866381380340
      2024-02-11T08:01:56.665-08:00 [Info] Indexer::handleMergePartition Source 12271221548839648418 Target 4054995742004854515
      panic: Inconsistent Internal State
       
      goroutine 2048870 [running]:
      github.com/couchbase/indexing/secondary/common.CrashOnError(...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:509
      github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).destTokenToMergeOrReady(0xc006740000, 0x1e02e8020026caf4, 0x13e25e0?, {0xc024bd8914?, 0xc04fa224e0?}, 0x1000000000019?, 0xc046aab7a0?)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2673 +0x28e
      github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).updateRStateToActive.func1(0x2)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2757 +0x225
      created by github.com/couchbase/indexing/secondary/indexer.(*ShardRebalancer).updateRStateToActive in goroutine 2048449
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/shard_rebalancer.go:2753 +0x154
      2024-02-11T08:01:58.042-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=1c901a3fe4a3d37e82947ba9a4a06a76 -isEnterprise=true --deploymentModel=default]
      

      One caveat is that the cluster has fewer index nodes and thus it might be undersized. At some point of the test, the cluster has ended up with 3 index nodes as opposed to 5 or 6. I'n unsure if that has anything to do with the panic. This was seen an hour or so after the rebalance failure reported in https://issues.couchbase.com/browse/MB-60758

      cbcollect ->

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.67.zip

      rl : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707669422/collectinfo-2024-02-11T164427-ns_1%40172.23.97.67.zip

      Older logs (n-1) ->

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707665544/collectinfo-2024-02-11T153705-ns_1%40172.23.97.67.zip

      Older logs (n-2)

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707661502/collectinfo-2024-02-11T143225-ns_1%40172.23.97.67.zip

      Older logs (n-3)

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707657466/collectinfo-2024-02-11T132503-ns_1%40172.23.97.67.zip

      Older logs (n-4)

      Cbcollect logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1707653307/collectinfo-2024-02-11T121747-ns_1%40172.23.97.67.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty