Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53207

[BP to 6.6.6 of MB-47655] feed gen-server crashed: runtime error: invalid memory address or nil pointer dereference

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Steps to reproduce:

      1. Create a 6 node cluster 3KV-2index-1query.
      2. Create about 7 buckets and indexes with replica.
      3. Load the docs until all of the buckets have data greater than memory.
      4. Kill memcached (by kill -9) and check that 1 KV node auto fails over.
      5. Do a full recovery and rebalance.

      Honestly I don't know which of these steps I was doing at the point of the panic as I only found out about it from the log_parser run that Pavan PB had started.

      172.23.100.34

      2022-08-01T05:39:40.858-07:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_eb12e1aeafb2cb873a3497fa53119e82(127.0.0.1:8091)] feed gen-server crashed: runtime error: invalid memory address or nil pointer dereference
      2022-08-01T05:39:40.858-07:00 [Error]   /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
      github.com/couchbase/indexing/secondary/projector.(*Feed).genServer.func1(0xc00168ac00)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:484 +0x124
      panic(0xbea400, 0x16e01d0)
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:679 +0x1b2
      github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0x0, 0xc037f7a387, 0x5, 0x0, 0x0, 0xcb3ba0, 0xc0497a8000, 0xc0007acf98, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:315 +0x86
      github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:444
      github.com/couchbase/indexing/secondary/dcp.(*Pool).getTerseBucket(0xc000430568, 0xc037f7a387, 0x5, 0x7f6da307a460, 0x0, 0x4f77796741667a6e, 0x3035693163707048)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:585 +0x14c
      github.com/couchbase/indexing/secondary/dcp.(*Pool).RefreshBucket(0xc000430568, 0xc037f7a387, 0x5, 0x7f6da307a400, 0x0, 0x30)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:605 +0x147
      github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchBucketInfo(0xc000430480, 0xc037f7a387, 0x5, 0x0, 0x0)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:385 +0x9f
      github.com/couchbase/indexing/secondary/projector.(*Feed).getLocalVbuckets(0xc00168ac00, 0xc037f7a380, 0x7, 0xc037f7a387, 0x5, 0x51, 0x0, 0x0, 0x0, 0x0, ...)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:1666 +0xad
      github.com/couchbase/indexing/secondary/projector.(*Feed).shutdownVbuckets(0xc00168ac00, 0xc042789500, 0xc038030051, 0xc0007ad630, 0x4174f3)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:1071 +0x124
      github.com/couchbase/indexing/secondary/projector.(*Feed).handleCommand(0xc00168ac00, 0xc042789f80, 0x4, 0x4, 0xcfd101, 0x2)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:706 +0x1792
      github.com/couchbase/indexing/secondary/projector.(*Feed).genServer(0xc00168ac00)
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:503 +0x2437
      created by github.com/couchbase/indexing/secondary/projector.NewFeed
              /home/couchbase/jenkins/workspace/toy-unix/goproj/src/github.com/couchbase/indexing/secondary/projector/feed.go:134 +0x72c
      

      cbcollect_info attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            yash.dodderi Yash Dodderi added a comment -

            i tried reproducing the above issue but was not able to. Had spoken to Sai Krishna Teja about this, he mentioned it is a race condition and it will be difficult to reproduce the issue and hence i would like to request dev verification of the same

            yash.dodderi Yash Dodderi added a comment - i tried reproducing the above issue but was not able to. Had spoken to Sai Krishna Teja about this, he mentioned it is a race condition and it will be difficult to reproduce the issue and hence i would like to request dev verification of the same

            Hi Yash Dodderi,

            This bug is originally found in QE testing. Please work with Balakumaran Gopal to get more info on repro steps.

            CC Hemant Rajput, Pavan PB.

            amit.kulkarni Amit Kulkarni added a comment - Hi Yash Dodderi , This bug is originally found in QE testing. Please work with Balakumaran Gopal to get more info on repro steps. CC Hemant Rajput , Pavan PB .

            Sai Krishna Teja and Amit Kulkarni, I've discussed this with Bala and he confirmed that this issue can't be reproduced consistently. We have already tried it couple of time to reproduce the issue, but we can't. Request you to close it on dev-verification.

             

            hemant.rajput Hemant Rajput added a comment - Sai Krishna Teja  and Amit Kulkarni , I've discussed this with Bala and he confirmed that this issue can't be reproduced consistently. We have already tried it couple of time to reproduce the issue, but we can't. Request you to close it on dev-verification.  
            amit.kulkarni Amit Kulkarni added a comment - Thanks Hemant Rajput .
            sai.teja Sai Krishna Teja added a comment - - edited

            Tested using a simple functional test with error injection on the second call. Saw a panic without the change

            func TestClusterInfoCache1(t *testing.T) {
                cic, err := c.NewClusterInfoCache("http://127.0.0.1:9001", c.DEFAULT_POOL)
                if err != nil {
                    t.Fatal(err)
                }       cic.Fetch()
                couchbase.TestErrorInject = 1
                cic.Fetch()
                cic.FetchBucketInfo("default")
            } 

            var TestErrorInject = 0
             
            // GetPool gets a pool from within the couchbase cluster (usually
            // "default").
            func (c *Client) GetPool(name string) (p Pool, err error) {
                if TestErrorInject != 0 {
                    var p Pool
                    return p, fmt.Errorf("Error Injected")
                } 
            ...

            Panic

            --- FAIL: TestClusterInfoCache1 (8.13s)
            panic: runtime error: invalid memory address or nil pointer dereference [recovered]
                panic: runtime error: invalid memory address or nil pointer dereference
            [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x43f3b42]goroutine 148 [running]:
            testing.tRunner.func1.2({0x49348a0, 0x52de4c0})
                /usr/local/go/src/testing/testing.go:1389 +0x24e
            testing.tRunner.func1()
                /usr/local/go/src/testing/testing.go:1392 +0x39f
            panic({0x49348a0, 0x52de4c0})
                /usr/local/go/src/runtime/panic.go:838 +0x207
            github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0x0, {0x4a599ef, 0x7}, {0x4a599ef?, 0x0?}, {0x4a09fa0, 0xc00b8b8900}, 0x0?)
                /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:315 +0xc2
            github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)
                /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:444
            github.com/couchbase/indexing/secondary/dcp.(*Pool).getTerseBucket(0xc004f3ac28, {0x4a599ef, 0x7})
                /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:585 +0xeb
            github.com/couchbase/indexing/secondary/dcp.(*Pool).RefreshBucket(0xc004f3ac28, {0x4a599ef, 0x7}, 0xc0?)
                /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:605 +0x136
            github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchBucketInfo(0xc004f3ab40, {0x4a599ef, 0x7})
                /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:385 +0x88
            github.com/couchbase/indexing/secondary/tests/functionaltests.TestClusterInfoCache1(0x61343522203a2264?)
                /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:236 +0xbb
            testing.tRunner(0xc005ca84e0, 0x4abbb40)
                /usr/local/go/src/testing/testing.go:1439 +0x102
            created by testing.(*T).Run
                /usr/local/go/src/testing/testing.go:1486 +0x35f
            exit status 2
            FAIL    github.com/couchbase/indexing/secondary/tests/functionaltests    117.944s

            With the change

            === RUN   TestClusterInfoCache1
            --- PASS: TestClusterInfoCache1 (8.10s)
            PASS
            ok      github.com/couchbase/indexing/secondary/tests/functionaltests    93.587s

             

            sai.teja Sai Krishna Teja added a comment - - edited Tested using a simple functional test with error injection on the second call. Saw a panic without the change func TestClusterInfoCache1(t *testing.T) {     cic, err := c.NewClusterInfoCache( "http://127.0.0.1:9001" , c.DEFAULT_POOL)     if err != nil {         t.Fatal(err)     }    cic.Fetch()     couchbase.TestErrorInject = 1     cic.Fetch()     cic.FetchBucketInfo( "default" ) } var TestErrorInject = 0   // GetPool gets a pool from within the couchbase cluster (usually // "default"). func (c *Client) GetPool(name string) (p Pool, err error) {     if TestErrorInject != 0 {         var p Pool         return p, fmt.Errorf( "Error Injected" )     } ... Panic --- FAIL: TestClusterInfoCache1 (8.13s) panic: runtime error: invalid memory address or nil pointer dereference [recovered]     panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x43f3b42]goroutine 148 [running]: testing.tRunner.func1.2({0x49348a0, 0x52de4c0})     /usr/local/go/src/testing/testing.go:1389 +0x24e testing.tRunner.func1()     /usr/local/go/src/testing/testing.go:1392 +0x39f panic({0x49348a0, 0x52de4c0})     /usr/local/go/src/runtime/panic.go:838 +0x207 github.com/couchbase/indexing/secondary/dcp.queryRestAPIOnLocalhost(0x0, {0x4a599ef, 0x7}, {0x4a599ef?, 0x0?}, {0x4a09fa0, 0xc00b8b8900}, 0x0?)     /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:315 +0xc2 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...)     /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:444 github.com/couchbase/indexing/secondary/dcp.(*Pool).getTerseBucket(0xc004f3ac28, {0x4a599ef, 0x7})     /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:585 +0xeb github.com/couchbase/indexing/secondary/dcp.(*Pool).RefreshBucket(0xc004f3ac28, {0x4a599ef, 0x7}, 0xc0?)     /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:605 +0x136 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchBucketInfo(0xc004f3ab40, {0x4a599ef, 0x7})     /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:385 +0x88 github.com/couchbase/indexing/secondary/tests/functionaltests.TestClusterInfoCache1(0x61343522203a2264?)     /Users/sai.teja/Repos/mad_hatter/top/goproj/src/github.com/couchbase/indexing/secondary/tests/functionaltests/common_test.go:236 +0xbb testing.tRunner(0xc005ca84e0, 0x4abbb40)     /usr/local/go/src/testing/testing.go:1439 +0x102 created by testing.(*T).Run     /usr/local/go/src/testing/testing.go:1486 +0x35f exit status 2 FAIL    github.com/couchbase/indexing/secondary/tests/functionaltests    117.944s With the change === RUN   TestClusterInfoCache1 --- PASS: TestClusterInfoCache1 (8.10s) PASS ok      github.com/couchbase/indexing/secondary/tests/functionaltests    93.587s  

            People

              sai.teja Sai Krishna Teja
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty