Details
-
Bug
-
Resolution: Fixed
-
Major
-
6.6.3
-
Untriaged
-
1
-
Unknown
Description
Build: 6.6.3-9796
Test: -test tests/fts/test_fts_alice_component.yml -scope tests/fts/scope_component_fts.yml
Scale: 1
Seeing below panic in 172.23.97.77:
2021-07-21T01:19:06.222-07:00 [INFO] cbdatasource: server: 172.23.104.17:11210, uprOpenName: fts:st_index_scorch_53a4c82fa840b526_aa574717-761d6b7c, worker, looping beg, vbucketState: "running" (has 68 vbuckets), 597-664
|
unexpected fault address 0x7ff4e3a9f669
|
fatal error: fault
|
[signal SIGBUS: bus error code=0x2 addr=0x7ff4e3a9f669 pc=0x402b3c]
|
|
goroutine 261196642 [running]:
|
runtime.throw(0xf68d06, 0x5)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/runtime/panic.go:608 +0x72 fp=0xc04277e7f8 sp=0xc04277e7c8 pc=0x42dfe2
|
runtime.sigpanic()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/runtime/signal_unix.go:387 +0x2d7 fp=0xc04277e848 sp=0xc04277e7f8 pc=0x4439d7
|
cmpbody()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.11.5/go/src/internal/bytealg/compare_amd64.s:124 +0xec fp=0xc04277e850 sp=0xc04277e848 pc=0x402b3c
|
github.com/couchbase/moss.(*segment).findKeyPos(0xc051775f00, 0xc01e270b08, 0x5, 0x6, 0xc04277e938, 0xb89786, 0xc0ba610280)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment.go:466 +0x2e6 fp=0xc04277e8e0 sp=0xc04277e850 pc=0xb86116
|
github.com/couchbase/moss.(*segment).Get(0xc051775f00, 0xc01e270b08, 0x5, 0x6, 0x20, 0xf, 0x1763a20, 0x175d940, 0x1, 0x1)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment.go:403 +0x50 fp=0xc04277e948 sp=0xc04277e8e0 pc=0xb85ca0
|
github.com/couchbase/moss.(*segmentStack).get(0xc0716ec0a0, 0xc01e270b08, 0x5, 0x6, 0x4, 0x0, 0xb90000, 0xc0378e6e40, 0x5, 0x5, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:93 +0x28a fp=0xc04277e9d8 sp=0xc04277e948 pc=0xb8922a
|
github.com/couchbase/moss.(*segmentStack).Get(0xc0716ec0a0, 0xc01e270b08, 0x5, 0x6, 0xc0716e0000, 0x4, 0x2, 0x1, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:78 +0x81 fp=0xc04277ea48 sp=0xc04277e9d8 pc=0xb88f41
|
github.com/couchbase/moss.(*Footer).Get(0xc0378e6e40, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x1e2, 0xc05f322000, 0xc015d26000, 0xc04277eb98)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/store_footer.go:465 +0x96 fp=0xc04277eae8 sp=0xc04277ea48 pc=0xb96c16
|
github.com/couchbase/moss.(*SnapshotWrapper).Get(0xc1165037d0, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x0, 0xc0006d7101, 0xc04277ec50, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/wrap.go:95 +0x6e fp=0xc04277eb48 sp=0xc04277eae8 pc=0xb991be
|
github.com/couchbase/moss.(*SnapshotWrapper).Get(0xc116503800, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/wrap.go:95 +0x6e fp=0xc04277eba8 sp=0xc04277eb48 pc=0xb991be
|
github.com/couchbase/moss.(*segmentStack).get(0xc089391ea0, 0xc01e270b08, 0x5, 0x6, 0x0, 0x0, 0x200000, 0xc04277eca0, 0x42172b, 0xc000055270, ...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:114 +0xdd fp=0xc04277ec38 sp=0xc04277eba8 pc=0xb8907d
|
github.com/couchbase/moss.(*segmentStack).Get(0xc089391ea0, 0xc01e270b08, 0x5, 0x6, 0xc0eb930000, 0x42a07e, 0x4140b1, 0xf6794e, 0xc01c902008, 0xc04277ed20)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/couchbase/moss/segment_stack.go:78 +0x81 fp=0xc04277eca8 sp=0xc04277ec38 pc=0xb88f41
|
github.com/blevesearch/bleve/index/store/moss.(*Reader).Get(0xc022ff2ac0, 0xc01e270b08, 0x5, 0x6, 0x6, 0x6, 0xc01c902008, 0x4, 0xc00306df8a)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/godeps/src/github.com/blevesearch/bleve/index/store/moss/reader.go:29 +0x6c fp=0xc04277ed30 sp=0xc04277eca8 pc=0xc51a8c
|
github.com/blevesearch/bleve/index/upsidedown.newUpsideDownCouchTermFieldReader(0xc022ff2ae0, 0xc01e270b00, 0x2, 0x8, 0x1010013, 0x0, 0xc01b426940, 0xc04277ee58)
|
The last operation from the test is rebalance out of 172.23.121.165
2021-07-20T15:46:59.621-07:00, ns_orchestrator:0:info:message(ns_1@172.23.104.16) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.16','ns_1@172.23.104.17',
|
'ns_1@172.23.104.18','ns_1@172.23.104.19',
|
'ns_1@172.23.104.21','ns_1@172.23.104.23',
|
'ns_1@172.23.97.77','ns_1@172.23.97.82',
|
'ns_1@172.23.97.83'], EjectNodes = ['ns_1@172.23.121.165'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e2740d22cbc8ef20de998919cb67e955
|
2021-07-20T15:47:00.578-07:00, ns_storage_conf:0:info:message(ns_1@172.23.104.18) - Deleting old data files of bucket "other"
|
2021-07-20T15:47:00.578-07:00, ns_storage_conf:0:info:message(ns_1@172.23.104.18) - Deleting old data files of bucket "default"
|
2021-07-20T15:47:00.594-07:00, ns_rebalancer:0:info:message(ns_1@172.23.104.16) - Started rebalancing bucket other
|
2021-07-20T15:47:00.799-07:00, ns_memcached:0:info:message(ns_1@172.23.104.18) - Bucket "other" loaded on node 'ns_1@172.23.104.18' in 0 seconds.
|
2021-07-20T15:47:02.004-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.104.16) - Bucket "other" rebalance appears to be swap rebalance
|
2021-07-20T15:49:11.736-07:00, ns_rebalancer:0:info:message(ns_1@172.23.104.16) - Started rebalancing bucket default
|
2021-07-20T15:49:11.900-07:00, ns_memcached:0:info:message(ns_1@172.23.104.18) - Bucket "default" loaded on node 'ns_1@172.23.104.18' in 0 seconds.
|
2021-07-20T15:49:12.125-07:00, ns_memcached:0:info:message(ns_1@172.23.121.165) - Shutting down bucket "other" on 'ns_1@172.23.121.165' for deletion
|
2021-07-20T15:49:13.083-07:00, ns_vbucket_mover:0:info:message(ns_1@172.23.104.16) - Bucket "default" rebalance appears to be swap rebalance
|
2021-07-20T15:52:17.754-07:00, ns_memcached:0:info:message(ns_1@172.23.121.165) - Shutting down bucket "default" on 'ns_1@172.23.121.165' for deletion
|
2021-07-20T16:24:43.992-07:00, ns_cluster:1:info:message(ns_1@172.23.121.165) - Node 'ns_1@172.23.121.165' is leaving cluster.
|
2021-07-20T16:24:44.098-07:00, ns_orchestrator:0:info:message(ns_1@172.23.104.16) - Rebalance completed successfully.
|
Rebalance Operation Id = e2740d22cbc8ef20de998919cb67e955
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.16.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.17.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.18.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.19.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.104.23.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.97.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.97.82.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1626858061/collectinfo-2021-07-21T090106-ns_1%40172.23.97.83.zip
########## Cluster config ##################
|
###### fts : 7 ===== > [172.23.104.16:8091 172.23.104.18:8091 172.23.104.19:8091 172.23.104.21:8091 172.23.104.23:8091 172.23.121.165:8091 172.23.97.77:8091] ###########
|
###### kv : 1 ===== > [172.23.104.17:8091] ###########
|