Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45959

ForestDB: signal SIGSEGV: segmentation violation code=0x1 addr=0xff0 pc=0x7fdbdfc6222e

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build : 7.0.0-5037 - Couchbase community Edition
      Storage Mode : ForestDB
      Test Script & Scope : -test tests/2i/cheshirecat/test_idx_cc_vol_ce.yml -scope tests/2i/cheshirecat/scope_idx_cc_vol_ce.yml
      Scale : 2

      Cluster Config:

      • 11 nodes, 24 GB RAM, kv quota: 11 GB , index quota: 9 GB
      • Kv,query,index on all nodes

      Some numbers from the test:

      • Buckets : 5
      • Scopes : 25
      • Collections : 100
      • Total number of index instances across the cluster : 1000
      • Avg. # docs per collection : 20000
      • KV+Index+query nodes in cluster : 11-12

      Test steps:

      • After creation of indexes performed below operations while queries being run in parallel
      • Rebalance in node with Kv,query,index
      • Rebalance out
      • Swap rebalance with Kv,query,index
      • Failover/Hard Failover, Full/Partial recover
      • Item count check between index and collection
      • Drop all indexes at the end of the test.

      Seeing below panic while drop all indexes on nodes: 172.23.97.227, 172.23.97.232, 172.23.97.235

      2021-04-27T17:44:53.972-07:00 [Info] Indexer::removeIndexesFromStream Success Stream MAINT_STREAM KeyspaceId bucket1 SessionId 1
      		 
      fatal error: unexpected signal during runtime execution
      		 
      [signal SIGSEGV: segmentation violation code=0x1 addr=0xff0 pc=0x7fdbdfc6222e]
      		 
       
      		 
      runtime stack:
      		 
      runtime.throw(0x118fd34, 0x2a)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:774 +0x72
      		 
      runtime.sigpanic()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/signal_unix.go:378 +0x47c
      		 
       
      		 
      goroutine 4758159 [syscall]:
      		 
      runtime.cgocall(0xebc7f0, 0xc0145cd648, 0xc0145cd678)
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/cgocall.go:128 +0x5b fp=0xc0145cd618 sp=0xc0145cd5e0 pc=0x40894b
      		 
      github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_iterator_next(0x7fd99143ad60, 0x7fd900000000)
      		 
              _cgo_gotypes.go:858 +0x49 fp=0xc0145cd648 sp=0xc0145cd618 pc=0x9f9469
      		 
      github.com/couchbase/indexing/secondary/fdb.(*Iterator).Next.func1(0xc014553c50, 0x118669b)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/iterator.go:93 +0x5e fp=0xc0145cd688 sp=0xc0145cd648 pc=0xa0746e
      		 
      github.com/couchbase/indexing/secondary/fdb.(*Iterator).Next(0xc014553c50, 0x0, 0x0)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/iterator.go:93 +0x12c fp=0xc0145cd710 sp=0xc0145cd688 pc=0xa020dc
      		 
      github.com/couchbase/indexing/secondary/indexer.(*ForestDBIterator).Next(0xc016a10ab0)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_iterator.go:151 +0x55 fp=0xc0145cd780 sp=0xc0145cd710 pc=0xd219e5
      		 
      github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Iterate(0xc01202f300, 0x1369860, 0xc0154c0080, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x11d4cb0, 0xc00acfbdd0, ...)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot_reader.go:227 +0x21e fp=0xc0145cd8d0 sp=0xc0145cd780 pc=0xd3400e
      		 
      github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Range(0xc01202f300, 0x1369860, 0xc0154c0080, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0xc00acfbdd0, 0xfdd4c0, ...)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot_reader.go:187 +0xb8 fp=0xc0145cd940 sp=0xc0145cd8d0 pc=0xd33d08
      		 
      github.com/couchbase/indexing/secondary/indexer.scanSingleSlice(0xc00a866580, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x116acca, 0xb, 0xc00ae6d860, 0x1, ...)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:206 +0x233 fp=0xc0145cda58 sp=0xc0145cd940 pc=0xe10ab3
      		 
      github.com/couchbase/indexing/secondary/indexer.scanOne(0xc00a866580, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x116acca, 0xb, 0xc00ae6d860, 0x1, ...)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:148 +0x111 fp=0xc0145cdb68 sp=0xc0145cda58 pc=0xe10671
      		 
      github.com/couchbase/indexing/secondary/indexer.scatter(0xc00a866580, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x116acca, 0xb, 0xc00ae6d860, 0x1, ...)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:55 +0xc4 fp=0xc0145cdc18 sp=0xc0145cdb68 pc=0xe0fa44
      		 
      github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine(0xc009a38460, 0x0, 0x0)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:341 +0xdae fp=0xc0145cdf58 sp=0xc0145cdc18 pc=0xdf536e
      		 
      github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt.func1(0xc00ad61e40, 0xc0145e6bc0)
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:75 +0x38 fp=0xc0145cdfd0 sp=0xc0145cdf58 pc=0xce45e8
      		 
      runtime.goexit()
      		 
              /home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0145cdfd8 sp=0xc0145cdfd0 pc=0x464bb1
      		 
      created by github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt
      		 
              /tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:74 +0x66

      Test log snippet while drop all indexes:

      [root@localhost ~]# docker run -it sequoiatools/indexmanager -n 172.23.97.215 -o 8091 -u Administrator -p password -b bucket1 -a drop_all_indexes -v
      		 
      2021-04-27 17:39:17,703 - indexmanager - INFO - Setting Max Replica for this test to : 3
      		 
      2021-04-27 17:39:27,722 - indexmanager - INFO - ['`bucket1`.`scope_3`.`coll_5`', '`bucket1`.`scope_3`.`coll_4`', '`bucket1`.`scope_3`.`coll_3`', '`bucket1`.`scope_3`.`coll_2`', '`bucket1`.`scope_3`.`coll_1`', '`bucket1`.`scope_3`.`coll_0`', '`bucket1`.`scope_2`.`coll_4`', '`bucket1`.`scope_2`.`coll_3`', '`bucket1`.`scope_2`.`coll_2`', '`bucket1`.`scope_2`.`coll_1`', '`bucket1`.`scope_2`.`coll_0`', '`bucket1`.`scope_1`.`coll_2`', '`bucket1`.`scope_1`.`coll_1`', '`bucket1`.`scope_1`.`coll_0`', '`bucket1`.`scope_0`.`coll_5`', '`bucket1`.`scope_0`.`coll_4`', '`bucket1`.`scope_0`.`coll_3`', '`bucket1`.`scope_0`.`coll_2`', '`bucket1`.`scope_0`.`coll_1`', '`bucket1`.`scope_0`.`coll_0`', '`bucket1`.`_default`.`_default`']
      		 
      2021-04-27 17:39:27,722 - indexmanager - INFO - Starting to drop all indexes 
      2021-04-27 17:45:34,066 - indexmanager - INFO - Drop all indexes completed
      		 
      2021-04-27 17:45:34,504 - indexmanager - INFO - Validation completed
      		 
      [root@localhost ~]# docker run -it sequoiatools/indexmanager -n 172.23.97.215 -o 8091 -u Administrator -p password -b bucket2 -a drop_all_indexes -v
      		 
      2021-04-27 17:53:39,922 - indexmanager - INFO - Setting Max Replica for this test to : 3
      		 
      2021-04-27 17:53:49,932 - indexmanager - INFO - ['`bucket2`.`scope_3`.`coll_7`', '`bucket2`.`scope_3`.`coll_6`', '`bucket2`.`scope_3`.`coll_5`', '`bucket2`.`scope_3`.`coll_4`', '`bucket2`.`scope_3`.`coll_3`', '`bucket2`.`scope_3`.`coll_2`', '`bucket2`.`scope_3`.`coll_1`', '`bucket2`.`scope_3`.`coll_0`', '`bucket2`.`scope_2`.`coll_4`', '`bucket2`.`scope_2`.`coll_3`', '`bucket2`.`scope_2`.`coll_2`', '`bucket2`.`scope_2`.`coll_1`', '`bucket2`.`scope_2`.`coll_0`', '`bucket2`.`scope_1`.`coll_2`', '`bucket2`.`scope_1`.`coll_1`', '`bucket2`.`scope_1`.`coll_0`', '`bucket2`.`scope_0`.`coll_3`', '`bucket2`.`scope_0`.`coll_2`', '`bucket2`.`scope_0`.`coll_1`', '`bucket2`.`scope_0`.`coll_0`', '`bucket2`.`_default`.`_default`']
      		 
      2021-04-27 17:53:49,932 - indexmanager - INFO - Starting to drop all indexes 
      2021-04-27 17:59:59,018 - indexmanager - INFO - Drop all indexes completed
      		 
      2021-04-27 17:59:59,386 - indexmanager - INFO - Validation completed

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.2.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.4.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.215.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.216.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.227.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.232.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.235.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.236.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.237.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            srinath.duvuru Srinath Duvuru
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty