Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47776

panic in switchPhase

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Operating System:
      Centos 64-bit
    • Story Points:
      1
    • Is this a Regression?:
      Unknown

      Description

      In 6.6.3-9805 there is a panic in the query logs:

      _time=2021-08-05T12:04:49.055-07:00 _level=SEVERE _msg=panic: runtime error: invalid memory address or nil pointer dereference 
      _time=2021-08-05T12:04:49.055-07:00 _level=SEVERE _msg=request text: <ud>PREPARE SELECT * FROM `bucket-1` USE KEYS[$1];</ud>
      

      This panic is also present in 6.6.3-9806. In addition in 6.6.3-9806, in the ns_server error log there are errors indicating connection refused by query admin/stats endpoint. This results in the reported query throughput number being lower but in reality, there is no performance regression.

      http://cbmonitor.sc.couchbase.com/reports/html/?snapshot=iris_663-9805_access_af0e&snapshot=iris_663-9806_access_c76e

      9805 - http://perf.jenkins.couchbase.com/job/iris-multi-client/10740/ - 126314.0

      9806 - http://perf.jenkins.couchbase.com/job/iris-multi-client/10737/ - 83959.0

      Not sure how long the query panic has been present, but can try to find the initial offending build. Looks like there is no relevant commit in 9806 to cause this issue. Last god run was 9804 so probably caused by this commit in 9805 and the admin/stats issue is spurious side effect:

      http://changelog.build.couchbase.com/?product=couchbase-server&fromVersion=6.6.3&fromBuild=9804&toVersion=6.6.3&toBuild=9806&f_backup=on&f_cbas-core=on&f_go-couchbase=on

      https://github.com/couchbase/go-couchbase/commit/4ed93c0ce9e9db8b97f184dccbcd724d46bfde02

      https://issues.couchbase.com/browse/MB-47722

        Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.1.0-1121 contains query commit 3061626 with commit message:
          MB-47776 race between Done() and switchPhase()

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1121 contains query commit 3061626 with commit message: MB-47776 race between Done() and switchPhase()
          Hide
          korrigan.clark Korrigan Clark added a comment - - edited

          queued up verification runs:

          7.0.1-5991: http://perf.jenkins.couchbase.com/job/iris-multi-client/10938/ - No panic

          7.1.0-1121: http://perf.jenkins.couchbase.com/job/iris-multi-client/10939/ - No panic

          Marco Greco is there a 6.6.4 build to test? I don't see any yet...

          Show
          korrigan.clark Korrigan Clark added a comment - - edited queued up verification runs: 7.0.1-5991: http://perf.jenkins.couchbase.com/job/iris-multi-client/10938/  - No panic 7.1.0-1121: http://perf.jenkins.couchbase.com/job/iris-multi-client/10939/  - No panic Marco Greco  is there a 6.6.4 build to test? I don't see any yet...
          Hide
          marco.greco Marco Greco added a comment -

          Korrigan Clark it's just been merged in 6.6.4, give it an hour...
          But I did test in on a toy build on 6.6.3.

          Show
          marco.greco Marco Greco added a comment - Korrigan Clark it's just been merged in 6.6.4, give it an hour... But I did test in on a toy build on 6.6.3.
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.4-9901 contains query commit 581ecec with commit message:
          MB-47776 race between Done() and switchPhase()

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.4-9901 contains query commit 581ecec with commit message: MB-47776 race between Done() and switchPhase()
          Hide
          korrigan.clark Korrigan Clark added a comment -

          Awaiting 6.6.4 results.... other builds were clean

          6.6.4-9901: http://perf.jenkins.couchbase.com/job/iris-multi-client/10941/ - 

          Show
          korrigan.clark Korrigan Clark added a comment - Awaiting 6.6.4 results.... other builds were clean 6.6.4-9901: http://perf.jenkins.couchbase.com/job/iris-multi-client/10941/  - 

            People

            Assignee:
            korrigan.clark Korrigan Clark
            Reporter:
            korrigan.clark Korrigan Clark
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty