Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43872

Upgrade:6.5.1->7.0.0: panic invalid memory address: SIGSEGV: segmentation violation

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 6.5.1-6299 --> 7.0.0-4291

      Panic seen in the below test with job cen7-2i-plasma-set5-job1-upgrade-6-5-1_RED

      ./testrunner -i /tmp/centos_upgrade2i_6_5_1_red.ini -p upgrade_to=7.0.0-4291,initial_version=6.5.1-6299,get-cbcollect-info=False,get-coredumps=False -t gsi.upgrade_gsi.UpgradeSecondaryIndex.test_online_upgrade_with_rebalance,nodes_init=3,services_init=kv:n1ql-index-index,standard_buckets=1,dataset=default,scan_consistency=request_plus,groups=simple,init_nodes=False,gsi_type=forestdb,nodes_out=2,nodes_out_dist=index:2,nodes_in=2,services_in=index:2,before=create_index,disable_plasma_upgrade=True

      Steps in the test

      • Setup 3 node 6.5.1-6299 cluster with kv+n1ql, index , index services
      • create standard_bucket0 bucket and load 98784.0 docs
      • create and build below indexes

        * CREATE PRIMARY INDEX ON standard_bucket0  USING GSI
        * CREATE INDEX index_name_0 ON standard_bucket0(job_title) USING GSI  WITH {'defer_build': True}
        * QUERY BUILD INDEX on standard_bucket0(index_name_0) USING GSI
        * QUERY CREATE INDEX employee1e2bf8807f254cc9ba1b61b874be5e7ajob_title ON standard_bucket0(job_title) WHERE  job_title IS NOT NULL  USING GSI  WITH {'nodes': ['172.23.105.15:8091'], 'defer_build': True}
        * QUERY CREATE INDEX employee1e2bf8807f254cc9ba1b61b874be5e7ajoin_yr ON standard_bucket0(join_yr) WHERE  join_yr > 2010 and join_yr < 2014  USING GSI  WITH {'nodes': ['172.23.105.18:8091'], 'defer_build': True}
        * QUERY BUILD INDEX on standard_bucket0(employee1e2bf8807f254cc9ba1b61b874be5e7ajob_title,employee1e2bf8807f254cc9ba1b61b874be5e7ajoin_yr) USING GSI
        

      • Start mutations on standard_bucket0
      • Parallely rebalance in index node with 7.0.0-4291

      Rebalance failed and below Panic seen 7.0.0-4291 indexer logs. 172.23.105.21 is the indexer with panic

      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x841947]
       
      goroutine 176 [running]:
      github.com/couchbase/indexing/secondary/common/collections.(*CollectionManifest).GetCollectionID(0x0, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0xc0069c8f01, 0xc0069c8f00)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/collections/collection_defs.go:24 +0x37
      github.com/couchbase/indexing/secondary/dcp.(*Pool).GetCollectionID(...)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:861
      github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).GetCollectionID(0xc000282000, 0xc007352c0e, 0x10, 0x125cd9c, 0x8, 0x125cd9c, 0x8, 0x8, 0xe)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:534 +0xb4
      github.com/couchbase/indexing/secondary/manager.(*IndexManager).getKeyspaceForCleanup(0xc00236e090, 0x0, 0x0, 0x0, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:768 +0x250
      github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0xc00236e090, 0xc004772ba0)
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:725 +0x12c
      created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal
              /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:213 +0xab4
      2021-01-24T04:30:52.796-08:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=98257c32e8df65177c5232d6189302e3 -ipv6=false -isEnterprise=true]
      

      Attachments

        Activity

          People

            hemant.rajput Hemant Rajput
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty