Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48653

BP[7.0.2] - Magma GSI index test hit "panic: runtime error: invalid memory address or nil pointer dereference"

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      The Magma GSI index run failed while creating index. There are 4 kv nodes and 1 indexer node. We load 10B docs with 1 replica. The build used in this run is 7.1.0-1273.

      panic: runtime error: invalid memory address or nil pointer dereferencepanic: runtime error: invalid memory address or nil pointer dereference[signal SIGSEGV: segmentation violation code=0x1 addr=0x178 pc=0xfe7f7a]
      goroutine 661526 [running]:github.com/couchbase/indexing/secondary/indexer.(*storageMgr).getIndexStorageStats(0xc000619a20, 0x0, 0xc05573b7c8, 0x6c8f11, 0xc00b63c480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1603 +0x115agithub.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleGetIndexStorageStats.func1(0xc000619a20, 0x14e4a80, 0xc0068d5c60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1377 +0xadcreated by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).handleGetIndexStorageStats /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1370 +0x8aInitializing write barrier = 8000

      Logs:

      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-09-23T173144-ns_1%40172.23.97.129.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-09-23T173144-ns_1%40172.23.97.21.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-09-23T173144-ns_1%40172.23.97.22.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-09-23T173144-ns_1%40172.23.97.23.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-09-23T173144-ns_1%40172.23.97.24.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Bo-Chun Wang, Checked the logs, we can see in logs even when recovery has taken more than 18 mins we dont see stats call failing with storageManger.stats pointer nil. so  you can mark the issue as resolved.

            yogendra.acharya Yogendra Acharya added a comment - Bo-Chun Wang , Checked the logs, we can see in logs even when recovery has taken more than 18 mins we dont see stats call failing with storageManger.stats pointer nil. so  you can mark the issue as resolved.
            mihir.kamdar Mihir Kamdar added a comment -

            Yogendra Acharya since the same magma test is not available for 7.0.2, how else can we verify the fix in 7.0.2 ? is there a functional scenario at a smaller scale possible, or can this be closed based on some unit test ?

            mihir.kamdar Mihir Kamdar added a comment - Yogendra Acharya since the same magma test is not available for 7.0.2, how else can we verify the fix in 7.0.2 ? is there a functional scenario at a smaller scale possible, or can this be closed based on some unit test ?

            There is no functional scenario at smaller scale, the issue is not specific to magma but its related to large index recovery. It won't happen with smaller indexes. During indexer warmup if first index recovery takes long time (approx 20 mins) then only this can be seen. To confirm if we can add such test I will still check with team.

            yogendra.acharya Yogendra Acharya added a comment - There is no functional scenario at smaller scale, the issue is not specific to magma but its related to large index recovery. It won't happen with smaller indexes. During indexer warmup if first index recovery takes long time (approx 20 mins) then only this can be seen. To confirm if we can add such test I will still check with team.

            ified by adding sleep(20 mins) in indexer warmup code.

            yogendra.acharya Yogendra Acharya added a comment - ified by adding sleep(20 mins) in indexer warmup code.
            mihir.kamdar Mihir Kamdar added a comment -

            Closed as per Yogendra's comments.

            mihir.kamdar Mihir Kamdar added a comment - Closed as per Yogendra's comments.

            People

              yogendra.acharya Yogendra Acharya
              yogendra.acharya Yogendra Acharya
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty