Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-43875

[BP to 6.5.2] Projector crashed while deleting buckets

    XMLWordPrintable

Details

    • Triaged
    • 1
    • Yes

    Description

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            sai.teja Sai Krishna Teja added a comment - Submitted the change in http://review.couchbase.org/c/indexing/+/144232  

            Girish Benakappa with 6.5.2-6623, we need to scan the logs for some jobs to ensure there is no panic in the projector logs.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Girish Benakappa with 6.5.2-6623, we need to scan the logs for some jobs to ensure there is no panic in the projector logs.
            girish.benakappa Girish Benakappa added a comment - - edited

            Sai Krishna Teja

            Is there a way to reproduce this in 6.5.2?
            I see in CC we hit this issue when logging warning from feed in projector log, when bucket not found.

            2021-01-24T12:01:18.626-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_fa7472d0e095d526fad07fc5251f5c74(127.0.0.1:8091)] feed gen-server crashed: runtime error: invalid memory address or nil pointer dereference
            2021-01-24T12:01:18.626-08:00 [Error]   /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22
            github.com/couchbase/indexing/secondary/projector.(*Feed).genServer.func1(0xc008472120)
            

            But in 6.5.2 , i do not see we logging anything from FEED. is there a way or a flag to log these messages? or am i missing anything?

            girish.benakappa Girish Benakappa added a comment - - edited Sai Krishna Teja Is there a way to reproduce this in 6.5.2? I see in CC we hit this issue when logging warning from feed in projector log, when bucket not found. 2021-01-24T12:01:18.626-08:00 [Error] FEED[<=>MAINT_STREAM_TOPIC_fa7472d0e095d526fad07fc5251f5c74(127.0.0.1:8091)] feed gen-server crashed: runtime error: invalid memory address or nil pointer dereference 2021-01-24T12:01:18.626-08:00 [Error] /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:312 +0x22 github.com/couchbase/indexing/secondary/projector.(*Feed).genServer.func1(0xc008472120) But in 6.5.2 , i do not see we logging anything from FEED. is there a way or a flag to log these messages? or am i missing anything?

            Hi Girish Benakappa

            That error will be printed when there is a crash. You should see the below prints without the crash as it is fixed now.

            logging.Warnf("cluster_info: Out of sync for bucket %s. Retrying to getTerseBucket. retry count %v", bucketn, retryCount)

            Also this should be fixed in CC after MB-43867 you should not see the crash and hence this error message should not bee seen.

            Thanks

            Sai

            sai.teja Sai Krishna Teja added a comment - Hi Girish Benakappa That error will be printed when there is a crash. You should see the below prints without the crash as it is fixed now. logging.Warnf("cluster_info: Out of sync for bucket %s. Retrying to getTerseBucket. retry count %v", bucketn, retryCount) Also this should be fixed in CC after  MB-43867  you should not see the crash and hence this error message should not bee seen. Thanks Sai

            Hi Sai Krishna Teja

            We have tried below tests:

            Job: centos-query_in_list_6.5_P1
            Job: centos-2i_basic-integration-memdb, test: test_delete_bucket_while_index_build

            Could not reproduce this issue with 6.5.2-6621. But the tests found MB-43867 in CC.

            As discussed, since its racy, please verify and close this issue with code instrumentation. Thanks

            girish.benakappa Girish Benakappa added a comment - Hi Sai Krishna Teja We have tried below tests: Job: centos-query_in_list_6.5_P1 Job: centos-2i_basic-integration-memdb, test: test_delete_bucket_while_index_build Could not reproduce this issue with 6.5.2-6621. But the tests found MB-43867 in CC. As discussed, since its racy, please verify and close this issue with code instrumentation. Thanks

            Hardcoded the bucket name to "" in refreshBuckets function. When creating index i am seeing the below logs. Not seeing a crash when logging the message.

            2021-02-03T12:11:01.803+05:30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 1
            2021-02-03T12:11:01.811+05:30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 2
            2021-02-03T12:11:01.818+05:30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 3
            2021-02-03T12:11:01.825+05:30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 4
            2021-02-03T12:11:01.832+05:30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 5
            

            sai.teja Sai Krishna Teja added a comment - Hardcoded the bucket name to "" in refreshBuckets function. When creating index i am seeing the below logs. Not seeing a crash when logging the message. 2021 - 02 -03T12: 11 : 01.803 + 05 : 30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 1 2021 - 02 -03T12: 11 : 01.811 + 05 : 30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 2 2021 - 02 -03T12: 11 : 01.818 + 05 : 30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 3 2021 - 02 -03T12: 11 : 01.825 + 05 : 30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 4 2021 - 02 -03T12: 11 : 01.832 + 05 : 30 [Warn] cluster_info: Out of sync for bucket . Retrying to getTerseBucket. retry count 5

            People

              sai.teja Sai Krishna Teja
              sai.teja Sai Krishna Teja
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty