Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48746

Indexer bug when running n1ql tests on AWS EC2

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Neo
    • Neo
    • storage-engine
    • AWS EC2 instances, both ARM (m6g.large) and x86 (m5.large).
    • 1

    Description

      When running n1ql throughput tests on AWS EC2 instances, both arm and x86 the test hangs on an unsupported media error (attatched).

      When opening the cluster on couchbase server ui it appears to be a bug related to the indexing, this is specifically affecting n1ql tests (attatched). The full logs continually attempt these same runs, primarily with RunMemQuotaTuner() and always in relation to the indexer.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          jliang John Liang added a comment -

          Please get cbcollect and provide the build version

          jliang John Liang added a comment - Please get cbcollect and provide the build version
          sean.corrigan Sean Corrigan added a comment - Build number: 7.1.0-1248   Full logs: https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T191715-ns_1%40ec2-18-204-43-53.compute-1.amazonaws.com.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T191715-ns_1%40ec2-3-237-49-12.compute-1.amazonaws.com.zip https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T191715-ns_1%40ec2-44-197-241-124.compute-1.amazonaws.com.zip
          jliang John Liang added a comment -

          Sean Corrigan Are these logs collected for ARM or x86?

          jliang John Liang added a comment - Sean Corrigan Are these logs collected for ARM or x86?
          jliang John Liang added a comment -

          This is the actual panic

          unexpected fault address 0x4004858b5f
          fatal error: fault
          [signal SIGBUS: bus error code=0x1 addr=0x4004858b5f pc=0xbe1074]
           
          goroutine 1100 [running]:
          runtime.throw(0x1142a17, 0x5)
          	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400675c9a0 sp=0x400675c970 pc=0x43ab34
          runtime.sigpanic()
          	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400675c9e0 sp=0x400675c9a0 pc=0x452274
          github.com/couchbase/nitro/skiplist.(*Node).getNext(...)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110
          github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004ab6820, 0x40042c6000, 0x11bd808, 0x400675caf0, 0x4004ab6838, 0xc383ec)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400675ca60 sp=0x400675c9f0 pc=0xbe1074
          github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004ab6820, 0x40042c6000, 0x11bd808, 0x400675caf0, 0x4004ab6838, 0x64, 0xc04ef2a45e3a8b60, 0x2c7e717387)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400675caa0 sp=0x400675ca60 pc=0xbe0d88
          github.com/couchbase/plasma.(*Shard).FindInstance(0x40003d6000, 0x1, 0x0)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1660 +0x114 fp=0x400675cd50 sp=0x400675caa0 pc=0xc38484
          github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003d6000, 0x4004a79440, 0x3e, 0x101, 0x0, 0x0, 0x0)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1189 +0xd8 fp=0x400675ce10 sp=0x400675cd50 pc=0xc34998
          github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11bd838, 0x11bd810, 0x11bd848, 0x4004b56b00, 0x4004b56b10, ...)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:380 +0xb0 fp=0x400675d1c0 sp=0x400675ce10 pc=0xc19520
          github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11bd838, 0x11bd810, 0x11bd848, 0x4004b56b00, 0x4004b56b10, ...)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:321 +0x108 fp=0x400675d8d0 sp=0x400675d1c0 pc=0xc19008
          github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x400499d0c0, 0x400610ca80, 0x4005f70000, 0x4004b56ae0)
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:413 +0xb0 fp=0x400675dfb0 sp=0x400675d8d0 pc=0xe59c30
          runtime.goexit()
          	/home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400675dfb0 sp=0x400675dfb0 pc=0x471514
          created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores
          	/home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:410 +0x2eac
           
          
          

          jliang John Liang added a comment - This is the actual panic unexpected fault address 0x4004858b5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004858b5f pc=0xbe1074]   goroutine 1100 [running]: runtime.throw(0x1142a17, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400675c9a0 sp=0x400675c970 pc=0x43ab34 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400675c9e0 sp=0x400675c9a0 pc=0x452274 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004ab6820, 0x40042c6000, 0x11bd808, 0x400675caf0, 0x4004ab6838, 0xc383ec) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400675ca60 sp=0x400675c9f0 pc=0xbe1074 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004ab6820, 0x40042c6000, 0x11bd808, 0x400675caf0, 0x4004ab6838, 0x64, 0xc04ef2a45e3a8b60, 0x2c7e717387) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400675caa0 sp=0x400675ca60 pc=0xbe0d88 github.com/couchbase/plasma.(*Shard).FindInstance(0x40003d6000, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1660 +0x114 fp=0x400675cd50 sp=0x400675caa0 pc=0xc38484 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003d6000, 0x4004a79440, 0x3e, 0x101, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1189 +0xd8 fp=0x400675ce10 sp=0x400675cd50 pc=0xc34998 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11bd838, 0x11bd810, 0x11bd848, 0x4004b56b00, 0x4004b56b10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:380 +0xb0 fp=0x400675d1c0 sp=0x400675ce10 pc=0xc19520 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11bd838, 0x11bd810, 0x11bd848, 0x4004b56b00, 0x4004b56b10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:321 +0x108 fp=0x400675d8d0 sp=0x400675d1c0 pc=0xc19008 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x400499d0c0, 0x400610ca80, 0x4005f70000, 0x4004b56ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:413 +0xb0 fp=0x400675dfb0 sp=0x400675d8d0 pc=0xe59c30 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400675dfb0 sp=0x400675dfb0 pc=0x471514 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:410 +0x2eac  

          Sean Corrigan 

           The above panic seems to be on arm64

           For ARM64, The build 7.1.0-1248 does not have the necessary changes made for ARM64.

           For plasma, Commit: 67dcb85,   build: couchbase-server-7.1.0-1277

           For MOI,  Commit 985fb61, couchbase-server-7.1.0-1361

          Can you please re-test with 7.1.0-1361 or any later build and let us know.

          saptarshi.sen Saptarshi Sen added a comment - Sean Corrigan    The above panic seems to be on arm64  For ARM64, The build 7.1.0-1248 does not have the necessary changes made for ARM64.  For plasma, Commit: 67dcb85,    build: couchbase-server-7.1.0-1277  For MOI,  Commit 985fb61 , couchbase-server-7.1.0-1361 Can you please re-test with 7.1.0-1361 or any later build and let us know.

          Saptarshi Sen

          Hi, thank you for sorting this for us, have just run two jobs with that version on both arm and x86 and the bug is no longer present, thank you.

          sean.corrigan Sean Corrigan added a comment - Saptarshi Sen Hi, thank you for sorting this for us, have just run two jobs with that version on both arm and x86 and the bug is no longer present, thank you.

          People

            saptarshi.sen Saptarshi Sen
            sean.corrigan Sean Corrigan
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty