Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-59799

[Upgrade] : Service 'fts' exited with status 2 during swap rebalance post community to enterprise edition upgrade

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. Installed Couchbase Community Edition 7.2.2-6401 and created a 3 node couchbase cluster with the following services 
        1. Node 1 - 172.23.122.185 - kv, index,n1ql
        2. Node 2 - 172.23.121.67 - kv, index, n1ql, fts
        3. Node 3 - 172.23.122.189 - kv, index, n1ql, fts
      2. Created a couchstore bucket named default
      3. Created a fulltext-index index_default on 172.23.122.189
      4. Ran many fts queries using the fts index
      5. A new node 172.23.122.114 was installed and initialized with Couchbase Enterprise Edition  7.6.0-1813
      6. Started a swap rebalance on the cluster removing 172.23.122.185
      7. Cleaned and installed Couchbase Enterprise Edition  7.6.0-1813 on 172.23.122.185
      8. Started a swap rebalance on the cluster removing 172.23.122.189

      Rebalance failed with 'Service 'fts' exited with status 2.'

      2023-11-20T06:33:42.933-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.122.114) - Rebalance exited with reason {service_rebalance_failed,fts,                              {agent_died,<33749.4625.0>,                               {lost_connection,                                {'ns_1@172.23.122.185',shutdown}}}}.Rebalance Operation Id = 2194dc74c98ef08112cc2a995873bdcf2023-11-20T06:33:43.366-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/kvmux.go:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000805880, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc000034230)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc000034230)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x6?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 88    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:33:48.770-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000731a00, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc000493c70)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc000493c70)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000522e28?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 47    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:33:54.186-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages::138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000854e00, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004ae150)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004ae150)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000656b40?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 77    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:33:59.588-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc0008cf280, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc000456070)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc000456070)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x31?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 60    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:34:04.998-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000800000, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004ac0e0)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004ac0e0)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x6e6172546e49222c?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 30    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:34:10.412-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:
      		 
      github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc00064fc00, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004c6540)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004c6540)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000524e20?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 99    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:34:15.811-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/kvmux.go:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc0006a1980, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0004858f0)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0004858f0)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x0?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 83    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:34:21.229-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc000519780, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc0008b0230)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc0008b0230)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0xc000529560?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 94    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9
      		 
      2023-11-20T06:34:26.657-08:00, ns_log:0:info:message(ns_1@172.23.122.185) - Service 'fts' exited with status 2. Restarting. Messages:    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/kvmux.go:138 +0xe5github.com/couchbase/gocbcore/v10.(*configManagementComponent).OnNewConfig(0xc0004fbf00, 0x2235?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/configmanagement_component.go:108 +0x414github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).doLoop(0xc00051f1f0)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:281 +0x886github.com/couchbase/gocbcore/v10.(*baseHTTPConfigController).DoLoop(0xc00051f1f0)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/basehttpcfgcontroller.go:106 +0x18github.com/couchbase/gocbcore/v10.(*seedConfigController).Run(0x0?)    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/seedcfgcontroller.go:34 +0x16created by github.com/couchbase/gocbcore/v10.createAgent in goroutine 84    /home/couchbase/.cbdepscache/gomodcache/pkg/mod/github.com/couchbase/gocbcore/v10@v10.2.9/agent.go:421 +0x2cc9

       

       

      Similar issues are also observed with 7.1.x upgrade to 7.6.0, logs attached

       

      Test runner script to reproduce

      ./testrunner -i /data/workspace/debian-p0-upgrade-vset00-00-ce_ee-new-from-72x-2b/testexec.15448.ini -p skip_init_check_cbserver=true,released_upgrade_version=7.6.0-1813,get-cbcollect-info=false,data_analysis=True,initial_build_type=community,ce_to_ee=true,skip_log_scan=true,upgrade_test=true,get-cbcollect-info=True,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 -t upgrade.upgrade_tests.UpgradeTests.test_upgrade,items=20000,initial_version=7.2.0-5325,nodes_init=3,initialize_events=create_fts_index_query_compare,initial-services-setting=kv,index,n1ql-kv,n1ql,index,fts-kv,index,n1ql,fts,upgrade_services_in=same,after_events=rebalance_in-run_fts_query_and_compare,after_upgrade_services_in=kv,index,n1ql,fts,disable_HTP=True,upgrade_test=True,init_nodes=False

      Job name : debian-upgrade_ce_ee-new-from-72x-2b

      Job link : http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-1813/jenkins_logs/test_suite_executor/640751/

       

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-59999 FTS] - Panic on cb community edition - runtime error: index out of range

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty