Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-63236

Panic seen in indexer - handleCreateSnapshot - invalid memory address or nil pointer dereference

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Cypher
    • Cypher
    • secondary-index
    • 7.7.0-1088
    • Untriaged
    • 0
    • Unknown

    Description

      During the system test run, a panic was observed after (about 3 minutes) a rebalance failure ( inducing a process kill of indexer). The rebalance was a rebalance out operation (single node). Not sure if it has any relevance on the panic occurrence.

      Panic seen on 122 -

      2024-08-20T15:14:14.576-07:00 [Info] TK StreamBegin INIT_STREAM bucket3:scope_0:coll_0 1013 25586095899995 0 39897 172.23.97.66:8091. HWT [0-0,0].
      2024-08-20T15:14:14.667-07:00 [Info] TK StreamBegin INIT_STREAM bucket3:scope_0:coll_0 1017 234124105372185 0 39897 172.23.97.66:8091. HWT [0-0,0].
      2024-08-20T15:14:14.734-07:00 [Info] StreamState::adjustNonSnapAlignedVbs
      	KeyspaceId bucket3:scope_0:coll_0 StreamId INIT_STREAM Vbuckets [].
      2024-08-20T15:14:14.734-07:00 [Info] Timekeeper::repairStream Nothing to repair for Stream INIT_STREAM and KeyspaceId bucket3:scope_0:coll_0
      2024-08-20T15:14:15.734-07:00 [Info] timekeeper.repairMissingStreamBegin stream INIT_STREAM done
      2024-08-20T15:14:16.755-07:00 [Info] serviceChangeNotifier: received PoolChangeNotification
      2024-08-20T15:14:17.471-07:00 [Info] Timekeeper::setSnapshotType INIT_STREAM bucket3:scope_0:coll_0 setting snapshot type as DISK_SNAP due to BuildCompletionTS
      2024-08-20T15:14:17.510-07:00 [Info] plasmaSlice::OpenSnapshot SliceId 0 IndexInstId 5570128615443204469 PartitionId 1 Creating New Snapshot SnapshotInfo: count:142937 committed:true
      2024-08-20T15:14:17.510-07:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 5570128615443204469 PartitionId: 1 SliceId: 0 Crc64: 3699229607154508880 (SnapshotInfo: count:142937 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 38.737004ms SnapOpenDur 102.901µs
      2024-08-20T15:14:17.510-07:00 [Info] PlasmaSlice Slice Id 0, IndexInstId 5570128615443204469, PartitionId 1 SnapshotId 1 Creating recovery point ...
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x338 pc=0x1699dbc]
       
      goroutine 51938175 [running]:
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotForIndex(0xc005792ea0, 0x3, {0xc0e4bd00c0, 0x16}, 0xc03fcfc540?, 0xc073f33bf0, 0xc00888c5a0, 0x4d4d1276c4bb0975, 0xc07fbc6f70, 0xc0295f8e08, ...)
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:667 +0x4dc
      github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotWorker.func1({0xc02b69b6c8, 0x1, 0xc03fcfcba0?})
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:466 +0x139
      created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotWorker in goroutine 51938174
      	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:464 +0x165
      2024-08-20T15:14:20.789-07:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem --caFile=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem --clientCertFile=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem --clientKeyFile=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cluster=127.0.0.1:8091 -storageDir=/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=79c5524544440aa1c955826238e5ff55 -isEnterprise=true --deploymentModel=default]
      2024-08-20T15:14:20.790-07:00 [Info] Indexer DeploymentModel is set to: default
      2024-08-20T15:14:20.790-07:00 [Info] Setting ipv6=false
      2024-08-20T15:14:20.797-07:00 [Info] Indexer::NewIndexer Status Warmup
      2024-08-20T15:14:20.813-07:00 [Info] IndexerSettingsManager: Setting numSliceWriters to 8 cgroup.max_cpu_percent: 800 runtime.NumCPU: 800
      

      cbcollect ->

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.105.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.106.171.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.106.30.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.96.198.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.96.230.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.96.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.97.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.97.108.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.97.109.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.97.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1724192347/collectinfo-2024-08-20T222811-ns_1%40172.23.97.67.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            shivansh.rustagi Shivansh Rustagi
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty