Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-32714

[BP 6.0.1] query panics on ORDER_BY query with partitioned index

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 6.0.1
    • 5.5.2, 6.0.0, 6.5.0
    • secondary-index
    • hebe cluster
      OS: CentOS 7
      CPU: E5-2680 v3 (48 vCPU)
      Memory: 64GB
      Disk: Samsung PM863a

    Description

      I'm going through Pavel's legacy so  might miss some details or report already known issues.

      We have 4-node YCSB Workload E test on partitioned index we run since 5.5.
      And today I noticed that it never actually worked. The scan query returns panic error once in a while. Many more such errors under load but even singe query via UI Query Workbench fails occasionally.

      Reproduces only on multi node and if there is ORDER BY.

      Index: 
      CREATE PRIMARY INDEX primary_idx ON `bucket-1` PARTITION BY HASH(META().id);
       

      Query:
      SELECT RAW meta().id FROM `bucket-1` WHERE meta().id > %1 ORDER BY meta().id LIMIT %2

      Error:

      _msg=panic: runtime error: invalid memory address or nil pointer dereference

       

      request text:request text:<ud>PREPARE `a271558af22275b2d75217b6de9d952bd64d5841` FROM SELECT RAW meta().id FROM `bucket-1` WHERE meta().id >= $1 ORDER BY meta().id LIMIT $2</ud>
      stack:goroutine 563965 [running]:github.com/couchbase/query/execution.(*Context).Recover(0xc42129d500) goproj/src/github.com/couchbase/query/execution/context.go:519 +0xbcpanic(0xea9280, 0x195a130) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/panic.go:489 +0x2cfgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).analyzeOrderBy(0xc423b53a20, 0xc42878c3c0, 0x4, 0x4, 0x8, 0xc4222071e0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:1743 +0x2abgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).scatter(0xc423b53a20, 0xc42988dab8, 0xc4222071e0, 0xc42a8df380, 0x4, 0x4, 0xc426fd2440, 0x4, 0x4, 0xc426fd2460, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:480 +0x521github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).doScan(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc423b53a20, 0x0, 0x0, 0x0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1251 +0xacdgithub.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).Scan3Internal(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc422c4ab90, 0x1, 0x1, 0xc422140000, 0x0, 0x0, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1057 +0x31cgithub.com/couchbase/indexing/secondary/queryport/n1ql.(*secondaryIndex3).Scan3(0xc422618270, 0xc42868b260, 0x24, 0xc422c4ab78, 0x1, 0x1, 0x0, 0x0, 0x0, 0xa, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/n1ql/secondary_index.go:1125 +0x494github.com/couchbase/query/execution.(*IndexScan3).scan(0xc429682000, 0xc42129d500, 0xc42868b3e0, 0x0, 0x0) goproj/src/github.com/couchbase/query/execution/scan_index3.go:173 +0x535created by github.com/couchbase/query/execution.(*IndexScan3).RunOnce.func1 goproj/src/github.com/couchbase/query/execution/scan_index3.go:66 +0x227 goroutine 563965 [running]:github.com/couchbase/query/execution.(*Context).Recover(0xc42129d500) goproj/src/github.com/couchbase/query/execution/context.go:519 +0xbcpanic(0xea9280, 0x195a130) /home/couchbase/.cbdepscache/exploded/x86_64/go-1.8.5/go/src/runtime/panic.go:489 +0x2cfgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).analyzeOrderBy(0xc423b53a20, 0xc42878c3c0, 0x4, 0x4, 0x8, 0xc4222071e0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:1743 +0x2abgithub.com/couchbase/indexing/secondary/queryport/client.(*RequestBroker).scatter(0xc423b53a20, 0xc42988dab8, 0xc4222071e0, 0xc42a8df380, 0x4, 0x4, 0xc426fd2440, 0x4, 0x4, 0xc426fd2460, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/scatter.go:480 +0x521github.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).doScan(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc423b53a20, 0x0, 0x0, 0x0) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1251 +0xacdgithub.com/couchbase/indexing/secondary/queryport/client.(*GsiClient).Scan3Internal(0xc421ee0fc0, 0xbeab3cb8f301d25c, 0xc42868b260, 0x24, 0xc422c4ab90, 0x1, 0x1, 0xc422140000, 0x0, 0x0, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/client/client.go:1057 +0x31cgithub.com/couchbase/indexing/secondary/queryport/n1ql.(*secondaryIndex3).Scan3(0xc422618270, 0xc42868b260, 0x24, 0xc422c4ab78, 0x1, 0x1, 0x0, 0x0, 0x0, 0xa, ...) goproj/src/github.com/couchbase/indexing/secondary/queryport/n1ql/secondary_index.go:1125 +0x494github.com/couchbase/query/execution.(*IndexScan3).scan(0xc429682000, 0xc42129d500, 0xc42868b3e0, 0x0, 0x0) goproj/src/github.com/couchbase/query/execution/scan_index3.go:173 +0x535created by github.com/couchbase/query/execution.(*IndexScan3).RunOnce.func1 goproj/src/github.com/couchbase/query/execution/scan_index3.go:66 +0x227_time=2018-11-05T12:12:52.662-08:00 _level=SEVERE _msg=panic: runtime error: invalid memory address or nil pointer dereference

       

      Logs:

      Index. Data, Query on every node

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.204.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.205.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.206.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-hebe-2378/172.23.100.207.zip

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              prathibha Prathibha Bisarahalli (Inactive)
              deepkaran.salooja Deepkaran Salooja
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty