Details
-
Bug
-
Resolution: Unresolved
-
Major
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build : 7.0.0-5037 - Couchbase community Edition
Storage Mode : ForestDB
Test Script & Scope : -test tests/2i/cheshirecat/test_idx_cc_vol_ce.yml -scope tests/2i/cheshirecat/scope_idx_cc_vol_ce.yml
Scale : 2
Cluster Config:
- 11 nodes, 24 GB RAM, kv quota: 11 GB , index quota: 9 GB
- Kv,query,index on all nodes
Some numbers from the test:
- Buckets : 5
- Scopes : 25
- Collections : 100
- Total number of index instances across the cluster : 1000
- Avg. # docs per collection : 20000
- KV+Index+query nodes in cluster : 11-12
Test steps:
- After creation of indexes performed below operations while queries being run in parallel
- Rebalance in node with Kv,query,index
- Rebalance out
- Swap rebalance with Kv,query,index
- Failover/Hard Failover, Full/Partial recover
- Item count check between index and collection
- Drop all indexes at the end of the test.
Seeing below panic while drop all indexes on nodes: 172.23.97.227, 172.23.97.232, 172.23.97.235
2021-04-27T17:44:53.972-07:00 [Info] Indexer::removeIndexesFromStream Success Stream MAINT_STREAM KeyspaceId bucket1 SessionId 1
|
fatal error: unexpected signal during runtime execution
|
[signal SIGSEGV: segmentation violation code=0x1 addr=0xff0 pc=0x7fdbdfc6222e]
|
|
runtime stack:
|
runtime.throw(0x118fd34, 0x2a)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/panic.go:774 +0x72
|
runtime.sigpanic()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/signal_unix.go:378 +0x47c
|
|
goroutine 4758159 [syscall]:
|
runtime.cgocall(0xebc7f0, 0xc0145cd648, 0xc0145cd678)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/cgocall.go:128 +0x5b fp=0xc0145cd618 sp=0xc0145cd5e0 pc=0x40894b
|
github.com/couchbase/indexing/secondary/fdb._Cfunc_fdb_iterator_next(0x7fd99143ad60, 0x7fd900000000)
|
_cgo_gotypes.go:858 +0x49 fp=0xc0145cd648 sp=0xc0145cd618 pc=0x9f9469
|
github.com/couchbase/indexing/secondary/fdb.(*Iterator).Next.func1(0xc014553c50, 0x118669b)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/iterator.go:93 +0x5e fp=0xc0145cd688 sp=0xc0145cd648 pc=0xa0746e
|
github.com/couchbase/indexing/secondary/fdb.(*Iterator).Next(0xc014553c50, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/fdb/iterator.go:93 +0x12c fp=0xc0145cd710 sp=0xc0145cd688 pc=0xa020dc
|
github.com/couchbase/indexing/secondary/indexer.(*ForestDBIterator).Next(0xc016a10ab0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_iterator.go:151 +0x55 fp=0xc0145cd780 sp=0xc0145cd710 pc=0xd219e5
|
github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Iterate(0xc01202f300, 0x1369860, 0xc0154c0080, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x11d4cb0, 0xc00acfbdd0, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot_reader.go:227 +0x21e fp=0xc0145cd8d0 sp=0xc0145cd780 pc=0xd3400e
|
github.com/couchbase/indexing/secondary/indexer.(*fdbSnapshot).Range(0xc01202f300, 0x1369860, 0xc0154c0080, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0xc00acfbdd0, 0xfdd4c0, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_snapshot_reader.go:187 +0xb8 fp=0xc0145cd940 sp=0xc0145cd8d0 pc=0xd33d08
|
github.com/couchbase/indexing/secondary/indexer.scanSingleSlice(0xc00a866580, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x116acca, 0xb, 0xc00ae6d860, 0x1, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:206 +0x233 fp=0xc0145cda58 sp=0xc0145cd940 pc=0xe10ab3
|
github.com/couchbase/indexing/secondary/indexer.scanOne(0xc00a866580, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x116acca, 0xb, 0xc00ae6d860, 0x1, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:148 +0x111 fp=0xc0145cdb68 sp=0xc0145cda58 pc=0xe10671
|
github.com/couchbase/indexing/secondary/indexer.scatter(0xc00a866580, 0x1370600, 0xc00ad61e00, 0x1370540, 0x1e968f0, 0x3, 0x116acca, 0xb, 0xc00ae6d860, 0x1, ...)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_scatter.go:55 +0xc4 fp=0xc0145cdc18 sp=0xc0145cdb68 pc=0xe0fa44
|
github.com/couchbase/indexing/secondary/indexer.(*IndexScanSource).Routine(0xc009a38460, 0x0, 0x0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_pipeline.go:341 +0xdae fp=0xc0145cdf58 sp=0xc0145cdc18 pc=0xdf536e
|
github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt.func1(0xc00ad61e40, 0xc0145e6bc0)
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:75 +0x38 fp=0xc0145cdfd0 sp=0xc0145cdf58 pc=0xce45e8
|
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.13.7/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0145cdfd8 sp=0xc0145cdfd0 pc=0x464bb1
|
created by github.com/couchbase/indexing/secondary/pipeline.(*Pipeline).runIt
|
/tmp/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/pipeline/pipeline.go:74 +0x66
|
Test log snippet while drop all indexes:
[root@localhost ~]# docker run -it sequoiatools/indexmanager -n 172.23.97.215 -o 8091 -u Administrator -p password -b bucket1 -a drop_all_indexes -v
|
2021-04-27 17:39:17,703 - indexmanager - INFO - Setting Max Replica for this test to : 3
|
2021-04-27 17:39:27,722 - indexmanager - INFO - ['`bucket1`.`scope_3`.`coll_5`', '`bucket1`.`scope_3`.`coll_4`', '`bucket1`.`scope_3`.`coll_3`', '`bucket1`.`scope_3`.`coll_2`', '`bucket1`.`scope_3`.`coll_1`', '`bucket1`.`scope_3`.`coll_0`', '`bucket1`.`scope_2`.`coll_4`', '`bucket1`.`scope_2`.`coll_3`', '`bucket1`.`scope_2`.`coll_2`', '`bucket1`.`scope_2`.`coll_1`', '`bucket1`.`scope_2`.`coll_0`', '`bucket1`.`scope_1`.`coll_2`', '`bucket1`.`scope_1`.`coll_1`', '`bucket1`.`scope_1`.`coll_0`', '`bucket1`.`scope_0`.`coll_5`', '`bucket1`.`scope_0`.`coll_4`', '`bucket1`.`scope_0`.`coll_3`', '`bucket1`.`scope_0`.`coll_2`', '`bucket1`.`scope_0`.`coll_1`', '`bucket1`.`scope_0`.`coll_0`', '`bucket1`.`_default`.`_default`']
|
2021-04-27 17:39:27,722 - indexmanager - INFO - Starting to drop all indexes
|
2021-04-27 17:45:34,066 - indexmanager - INFO - Drop all indexes completed
|
2021-04-27 17:45:34,504 - indexmanager - INFO - Validation completed
|
[root@localhost ~]# docker run -it sequoiatools/indexmanager -n 172.23.97.215 -o 8091 -u Administrator -p password -b bucket2 -a drop_all_indexes -v
|
2021-04-27 17:53:39,922 - indexmanager - INFO - Setting Max Replica for this test to : 3
|
2021-04-27 17:53:49,932 - indexmanager - INFO - ['`bucket2`.`scope_3`.`coll_7`', '`bucket2`.`scope_3`.`coll_6`', '`bucket2`.`scope_3`.`coll_5`', '`bucket2`.`scope_3`.`coll_4`', '`bucket2`.`scope_3`.`coll_3`', '`bucket2`.`scope_3`.`coll_2`', '`bucket2`.`scope_3`.`coll_1`', '`bucket2`.`scope_3`.`coll_0`', '`bucket2`.`scope_2`.`coll_4`', '`bucket2`.`scope_2`.`coll_3`', '`bucket2`.`scope_2`.`coll_2`', '`bucket2`.`scope_2`.`coll_1`', '`bucket2`.`scope_2`.`coll_0`', '`bucket2`.`scope_1`.`coll_2`', '`bucket2`.`scope_1`.`coll_1`', '`bucket2`.`scope_1`.`coll_0`', '`bucket2`.`scope_0`.`coll_3`', '`bucket2`.`scope_0`.`coll_2`', '`bucket2`.`scope_0`.`coll_1`', '`bucket2`.`scope_0`.`coll_0`', '`bucket2`.`_default`.`_default`']
|
2021-04-27 17:53:49,932 - indexmanager - INFO - Starting to drop all indexes
|
2021-04-27 17:59:59,018 - indexmanager - INFO - Drop all indexes completed
|
2021-04-27 17:59:59,386 - indexmanager - INFO - Validation completed
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1619572871/collectinfo-2021-04-28T012112-ns_1%40172.23.97.237.zip