Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.6.0
-
Operating System : Debian GNU/Linux 10 (buster)
Initial Version : Couchbase Community Edition 7.2.2-6401
Upgrade Version : Couchbase Enterprise Edition 7.6.0-1813
-
Untriaged
-
-
0
-
Unknown
Description
Steps to reproduce
- Installed Couchbase Community Edition 7.2.2-6401 and created a 3 node couchbase cluster with the following services
-
- Node 1 - 172.23.122.185 - kv, index,n1ql
- Node 2 - 172.23.121.67 - kv, index, n1ql, fts
- Node 3 - 172.23.122.189 - kv, index, n1ql, fts
- Created a couchstore bucket named default
- Created a fulltext-index index_default on 172.23.122.189
- Ran many fts queries using the fts index
- A new node 172.23.122.114 was installed and initialized with Couchbase Enterprise Edition 7.6.0-1813
- Started a swap rebalance on the cluster removing 172.23.122.185
- Cleaned and installed Couchbase Enterprise Edition 7.6.0-1813 on 172.23.122.185
- Started a swap rebalance on the cluster removing 172.23.122.189
Rebalance failed with 'Service 'fts' exited with status 2.'
2023-11-20T06:33:42.933-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.122.114) - Rebalance exited with reason {service_rebalance_failed,fts, {agent_died,<33749.4625.0>, {lost_connection, {'ns_1@172.23.122.185',shutdown}}}}.Rebalance Operation Id = 2194dc74c98ef08112cc2a995873bdcf2023-11-20T06:33:43.366-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages: /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/kvmux.go:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000805880, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc000034230) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc000034230) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x6?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 88 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:33:48.770-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000731a00, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc000493c70) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc000493c70) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000522e28?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 47 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:33:54.186-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages::138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000854e00, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004ae150) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004ae150) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000656b40?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 77 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:33:59.588-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc0008cf280, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc000456070) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc000456070) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x31?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 60 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:34:04.998-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000800000, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004ac0e0) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004ac0e0) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x6e6172546e49222c?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 30 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:34:10.412-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages: |
github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc00064fc00, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004c6540) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004c6540) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000524e20?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 99 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:34:15.811-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages: /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/kvmux.go:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc0006a1980, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004858f0) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004858f0) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x0?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 83 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:34:21.229-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000519780, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0008b0230) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0008b0230) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000529560?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 94 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
2023-11-20T06:34:26.657-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages: /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/kvmux.go:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc0004fbf00, 0x2235?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc00051f1f0) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc00051f1f0) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x0?) /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 84 /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9 |
Similar issues are also observed with 7.1.x upgrade to 7.6.0, logs attached
./testrunner -i /data/workspace/debian-p0-upgrade-vset00-00-ce_ee-new-from-72x-2b/testexec.15448.ini -p skip_init_check_cbserver=true,released_upgrade_version=7.6.0-1813,get-cbcollect-info=false,data_analysis=True,initial_build_type=community,ce_to_ee=true,skip_log_scan=true,upgrade_test=true,get-cbcollect-info=True,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 -t upgrade.upgrade_tests.UpgradeTests.test_upgrade,items=20000,initial_version=7.2.0-5325,nodes_init=3,initialize_events=create_fts_index_query_compare,initial-services-setting=kv,index,n1ql-kv,n1ql,index,fts-kv,index,n1ql,fts,upgrade_services_in=same,after_events=rebalance_in-run_fts_query_and_compare,after_upgrade_services_in=kv,index,n1ql,fts,disable_HTP=True,upgrade_test=True,init_nodes=False
Job name : debian-upgrade_ce_ee-new-from-72x-2b
Attachments
Issue Links
- duplicates
-
MB-59999 FTS] - Panic on cb community edition - runtime error: index out of range
- Closed