Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43874

[2i] Panic after upgrade from 6.5.0 to 7.0.0 with partial indexes

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      I have not gotten a local repro, but on jenkins this job is semi-regularly failing due to panics.

      This test has the following steps:

      3 nodes
      1 node (kv,n1ql) - 172.23.123.64
      2 node(index) - 172.23.123.62 / 172.23.123.63

      1 bucket - standard_bucket0

      Before upgrade we create the following indexes:
      CREATE INDEX index_name_0 ON standard_bucket0(job_title) USING GSI WITH

      {'defer_build': True}

      BUILD INDEX on standard_bucket0(index_name_0) USING GSI
      CREATE INDEX idx_pending ON standard_bucket0(createdDateTime) WHERE status = 'PENDING'
      CREATE INDEX idx_success ON standard_bucket0(createdDateTime) WHERE status = 'SUCCESS'
      CREATE INDEX idx_error ON standard_bucket0(createdDateTime) WHERE status = 'ERROR'

      Then the test populates the following data:
      INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K1',

      {'createdDateTime': 4, 'status': 'PENDING'}

      )
      INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K2',

      {'createdDateTime': 5, 'status': 'PENDING'}

      )
      INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K3',

      {'createdDateTime': 6, 'status': 'PENDING'}

      )
      INSERT INTO `standard_bucket0` ( KEY, VALUE ) VALUES ('K4',

      {'createdDateTime': 7, 'status': 'PENDING'}

      )

      The test then updates the data:
      UPDATE `standard_bucket0` USE KEYS ['K1', 'K2', 'K3'] SET status = 'SUCCESS'
      UPDATE `standard_bucket0` USE KEYS 'K4' SET status = 'ERROR'

      After this the test then runs the upgrade to 7.0.0 by stopping the couchbase service and performing upgrade then bringing the nodes back online. The order is .64 (kv,n1ql), .62(index), .63(index)

      After upgrade is complete we see this error:

      [2021-01-24 09:03:02,720] - [rest_client:1021] ERROR - socket error while connecting to http://172.23.123.64:8091/indexStatus error [Errno 99] Cannot assign requested address

      And upon inspecting logs we see a panic on .63, will attach logs from all servers

      2021-01-24T08:58:02.013-08:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
      InstanceId: 12081943181962620740 Name: index_name_0 Keyspace: standard_bucket0:_default:_default State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x841947]

      goroutine 1 [running]:
      github.com/couchbase/indexing/secondary/common/collections.(*CollectionManifest).GetCollectionID(0x0, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0xc000105601, 0xc004c3e2c8)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/collections/collection_defs.go:24 +0x37
      github.com/couchbase/indexing/secondary/dcp.(*Pool).GetCollectionID(...)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:861
      github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).GetCollectionID(0xc000310900, 0xc004c28460, 0x10, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0x1, 0x2)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:534 +0xb4
      github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).ValidateCollectionID.func1(...)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:991
      github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).ValidateCollectionID(0xc00034a240, 0xc004c28460, 0x10, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0x125816f, 0x1, 0x100)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:998 +0xf3
      github.com/couchbase/indexing/secondary/indexer.(*indexer).validateIndexInstMap(0xc0012c0700)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7130 +0x575
      github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1(0xc0012c0700, 0xc000100b40, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6130 +0x3fb
      github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc00013d710, 0x26, 0x7ffc000001ed, 0x0, 0x0)
      /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:397 +0x1a71
      main.main()
      goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:169 +0x11c4
      2021-01-24T08:58:14.934-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=cad378e00fba89ff0284fe3dd4f47ad1 -ipv6=false -isEnterprise=true]

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jeelan.poola Jeelan Poola
            ajay.bhullar Ajay Bhullar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty