Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47549

panic: unexpected fault address 0x7ff4e3a9f669

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 6.6.3-9796
      Test: -test tests/fts/test_fts_alice_component.yml -scope tests/fts/scope_component_fts.yml
      Scale: 1

      Seeing below panic in 172.23.97.77:

      2021-07-21T01:19:06.222-07:00 [INFO] cbdatasource: server: 172.23.104.17:11210, uprOpenName: fts:st_index_scorch_53a4c82fa840b526_aa574717-761d6b7c, worker, looping beg, vbucketState: "running" (has 68 vbuckets), 597-664
      		 
      unexpected fault address 0x7ff4e3a9f669
      		 
      fatal error: fault
      		 
      [signal SIGBUS: bus error code=0x2 addr=0x7ff4e3a9f669 pc=0x402b3c]
      		 
       
      		 
      goroutine 261196642 [running]:
      		 
      runtime.throw(0xf68d06, 0x5)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/runtime/panic.go:608 +0x72 fp=0xc04277e7f8 sp=0xc04277e7c8 pc=0x42dfe2
      		 
      runtime.sigpanic()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/runtime/signal_unix.go:387 +0x2d7 fp=0xc04277e848 sp=0xc04277e7f8 pc=0x4439d7
      		 
      cmpbody()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/internal/bytealg/compare_amd64.s:124 +0xec fp=0xc04277e850 sp=0xc04277e848 pc=0x402b3c
      		 
      github.com/couchbase/moss.(*segment).findKeyPos(0xc051775f00, 0xc01e270b08, 0x5, 0x6, 0xc04277e938, 0xb89786, 0xc0ba610280)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment.go:466 +0x2e6 fp=0xc04277e8e0 sp=0xc04277e850 pc=0xb86116
      		 
      github.com/couchbase/moss.(*segment).Get(0xc051775f00, 0xc01e270b08, 0x5, 0x6, 0x20, 0xf, 0x1763a20, 0x175d940, 0x1, 0x1)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment.go:403 +0x50 fp=0xc04277e948 sp=0xc04277e8e0 pc=0xb85ca0
      		 
      github.com/couchbase/moss.(*segmentStack).get(0xc0716ec0a0, 0xc01e270b08, 0x5, 0x6, 0x4, 0x0, 0xb90000, 0xc0378e6e40, 0x5, 0x5, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:93 +0x28a fp=0xc04277e9d8 sp=0xc04277e948 pc=0xb8922a
      		 
      github.com/couchbase/moss.(*segmentStack).Get(0xc0716ec0a0, 0xc01e270b08, 0x5, 0x6, 0xc0716e0000, 0x4, 0x2, 0x1, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:78 +0x81 fp=0xc04277ea48 sp=0xc04277e9d8 pc=0xb88f41
      		 
      github.com/couchbase/moss.(*Footer).Get(0xc0378e6e40, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x1e2, 0xc05f322000, 0xc015d26000, 0xc04277eb98)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/store_footer.go:465 +0x96 fp=0xc04277eae8 sp=0xc04277ea48 pc=0xb96c16
      		 
      github.com/couchbase/moss.(*SnapshotWrapper).Get(0xc1165037d0, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x0, 0xc0006d7101, 0xc04277ec50, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/wrap.go:95 +0x6e fp=0xc04277eb48 sp=0xc04277eae8 pc=0xb991be
      		 
      github.com/couchbase/moss.(*SnapshotWrapper).Get(0xc116503800, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/wrap.go:95 +0x6e fp=0xc04277eba8 sp=0xc04277eb48 pc=0xb991be
      		 
      github.com/couchbase/moss.(*segmentStack).get(0xc089391ea0, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x200000, 0xc04277eca0, 0x42172b, 0xc000055270, ...)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:114 +0xdd fp=0xc04277ec38 sp=0xc04277eba8 pc=0xb8907d
      		 
      github.com/couchbase/moss.(*segmentStack).Get(0xc089391ea0, 0xc01e270b08, 0x5, 0x6, 0xc0eb930000, 0x42a07e, 0x4140b1, 0xf6794e, 0xc01c902008, 0xc04277ed20)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:78 +0x81 fp=0xc04277eca8 sp=0xc04277ec38 pc=0xb88f41
      		 
      github.com/blevesearch/bleve/index/store/moss.(*Reader).Get(0xc022ff2ac0, 0xc01e270b08, 0x5, 0x6, 0x6, 0x6, 0xc01c902008, 0x4, 0xc00306df8a)
      		 
              /home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/store/moss/reader.go:29 +0x6c fp=0xc04277ed30 sp=0xc04277eca8 pc=0xc51a8c
      		 
      github.com/blevesearch/bleve/index/upsidedown.newUpsideDownCouchTermFieldReader(0xc022ff2ae0, 0xc01e270b00, 0x2, 0x8, 0x1010013, 0x0, 0xc01b426940, 0xc04277ee58)

      The last operation from the test is rebalance out of 172.23.121.165

      2021-07-20T15:46:59.621-07:00, ns_orchestrator:0:info:message(ns_1@172.23.104.16) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.16','ns_1@172.23.104.17',
      		 
                                       'ns_1@172.23.104.18','ns_1@172.23.104.19',
      		 
                                       'ns_1@172.23.104.21','ns_1@172.23.104.23',
      		 
                                       'ns_1@172.23.97.77','ns_1@172.23.97.82',
      		 
                                       'ns_1@172.23.97.83'], EjectNodes = ['ns_1@172.23.121.165'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e2740d22cbc8ef20de998919cb67e955
      		 
      2021-07-20T15:47:00.578-07:00, ns_storage_conf:0:info:message(ns_1@172.23.104.18) - Deleting old data files of bucket "other"
      		 
      2021-07-20T15:47:00.578-07:00, ns_storage_conf:0:info:message(ns_1@172.23.104.18) - Deleting old data files of bucket "default"
      		 
      2021-07-20T15:47:00.594-07:00, ns_rebalancer:0:info:message(ns_1@172.23.104.16) - Started rebalancing bucket other
      		 
      2021-07-20T15:47:00.799-07:00, ns_memcached:0:info:message(ns_1@172.23.104.18) - Bucket "other" loaded on node 'ns_1@172.23.104.18' in 0 seconds.
      		 
      2021-07-20T15:47:02.004-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.104.16) - Bucket "other" rebalance appears to be swap rebalance
      		 
      2021-07-20T15:49:11.736-07:00, ns_rebalancer:0:info:message(ns_1@172.23.104.16) - Started rebalancing bucket default
      		 
      2021-07-20T15:49:11.900-07:00, ns_memcached:0:info:message(ns_1@172.23.104.18) - Bucket "default" loaded on node 'ns_1@172.23.104.18' in 0 seconds.
      		 
      2021-07-20T15:49:12.125-07:00, ns_memcached:0:info:message(ns_1@172.23.121.165) - Shutting down bucket "other" on 'ns_1@172.23.121.165' for deletion
      		 
      2021-07-20T15:49:13.083-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.104.16) - Bucket "default" rebalance appears to be swap rebalance
      		 
      2021-07-20T15:52:17.754-07:00, ns_memcached:0:info:message(ns_1@172.23.121.165) - Shutting down bucket "default" on 'ns_1@172.23.121.165' for deletion
      		 
      2021-07-20T16:24:43.992-07:00, ns_cluster:1:info:message(ns_1@172.23.121.165) - Node 'ns_1@172.23.121.165' is leaving cluster.
      		 
      2021-07-20T16:24:44.098-07:00, ns_orchestrator:0:info:message(ns_1@172.23.104.16) - Rebalance completed successfully.
      		 
      Rebalance Operation Id = e2740d22cbc8ef20de998919cb67e955

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.16.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.17.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.18.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.19.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.23.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.97.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.97.82.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.97.83.zip

      ########## Cluster config ##################
      		 
      ######  fts : 7 ===== > [172.23.104.16:8091 172.23.104.18:8091 172.23.104.19:8091 172.23.104.21:8091 172.23.104.23:8091 172.23.121.165:8091 172.23.97.77:8091]  ###########
      		 
      ######  kv : 1 ===== > [172.23.104.17:8091]  ###########

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            abhinav Abhi Dangeti
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty