Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60668

[Upgrade][Index] : Observing panic: HTTP error 503 Service Unavailable getting "http://127.0.0.1:8091/pools": "Temporary error occurred. Please try again later."

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • Morpheus, 7.6.2
    • 7.6.0
    • secondary-index
    • Operating System : Debian GNU/Linux 10 (buster)
      Initial Version : Couchbase Enterprise Edition build 7.2.2-6401
      Upgrade Version : Couchbase Enterprise Edition build 7.6.0-2095

    Description

      Steps to reproduce

      1. Created a 5 node cluster with Couchbase Enterprise Edition build 7.2.2-6401
        1. 172.23.121.218 - kv, index, n1ql
        2. 172.23.121.247 - cbas
        3. 172.23.121.248 - kv, index, n1ql
        4. 172.23.121.21 - cbas
        5. 172.23.121.228 - cbas
      2. Created a magma bucket named "bucket-3" and loaded 10000 documents onto it
      3. Created a few dataverses, links, datasets, synonyms, index and secondary indexes on cbas
      4. Added node 172.23.121.247 with Couchbase Enterprise Edition 7.6.0-2095
      5. Swap rebalanced out the node 172.23.121.203 

      Observing panics on 172.23.121.248 in ns_server.indexer.log

       

      2024-02-05T00:53:13.411-08:00 [Error] Indexer::NewIndexer Unable to get new ClusterInfoProvider err: HTTP error 503 Service Unavailable getting "http://127.0.0.1:8091/pools": "Temporary error occurred. Please try again later." use_cinfo_lite: truepanic: HTTP error 503 Service Unavailable getting "http://127.0.0.1:8091/pools": "Temporary error occurred. Please try again later."
      		 
      goroutine 1 [running]:github.com/couchbase/indexing/secondary/common.CrashOnError(...)    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:464github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x1f9de10?)    /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:380 +0x27b9main.main()    goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:182 +0xfadInitializing write barrier = 8000

       

      TAF Script to reproduce

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /data/workspace/debian-p0-analytics-vset00-00-analytics_upgrade_from_7.2.2_with_collections/testexec.30660.ini -p GROUP=7_2_2,kv_quota_percent=70,key=test_collections,get-cbcollect-info=True,upgrade_version=7.6.0-2095,aws_access_key=xxxxx,aws_secret_key=xxxxx,sirius_url=http://172.23.120.103:4000 -t upgrade.cbas_upgrade.UpgradeTests.test_upgrade,upgrade_chain=7.2.2,upgrade_type=online_rebalance_out_in,update_nodes=kv;cbas,nodes_init=5,services_init=kv:index:n1ql-kv:index:n1ql-cbas-cbas-cbas,pre_update_no_of_dv=2,pre_update_ds_per_dv=4,pre_update_no_of_synonym=5,pre_update_no_of_index=3,replica_num=3,override_spec_params=num_buckets;num_scopes;num_collections;replicas;num_items,num_items=10000,num_buckets=3,num_scopes=5,num_collections=5,no_of_dv=10,ds_per_dv=3,no_of_synonym=10,no_of_index=5,GROUP=7_2_2'

      Job name : debian-analytics-analytics_upgrade_from_7.2.2_with_collections

       

      Job ref : http://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-2095/jenkins_logs/test_suite_executor-TAF/309126/

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-60498 [Query] : Observing FATAL logs in query.log - [Fatal] MetakvGet: Failed to fetch /indexing/ddl/commandToken/delete/5848318864860926916 from metakv: ns_server _metakv returned: 503 Service Unavailable

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MB-60686 Add retries to "503 Service Unavailable" error on`ns_server` API calls

          • Major - Major loss of function.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              dhruvil.ketanshah Dhruvil Shah
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty