Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60498

[Query] : Observing FATAL logs in query.log - [Fatal] MetakvGet: Failed to fetch /indexing/ddl/commandToken/delete/5848318864860926916 from metakv: ns_server _metakv returned: 503 Service Unavailable

    XMLWordPrintable

Details

    Description

      Steps to reproduce

      1. On an ipv6 single node cluster, created a magma buckets named "default" and "standard_bucket0"
      2. Added 5 ipv6 nodes to the cluster
        1. fd63:6f75:6368:20d3:6000:b1ff:fea5:20d7 - kv, index, n1ql
        2. s10505-ip6.qe.couchbase.com - kv
        3. s10507-ip6.qe.couchbase.com - kv
        4. s10508-ip6.qe.couchbase.com - kv
        5. s10509-ip6.qe.couchbase.com - kv
        6. s10510-ip6.qe.couchbase.com - kv
      3. Rebalance completed successfully
      4. Deleted all buckets and ejected all nodes

      Observing FATAL and ERROR logs in query.logs on fd63:6f75:6368:20d3:6000:b1ff:fea5:20d7 or s10506-ip6.qe.couchbase.com

      2024-01-22T21:17:01.978-08:00 [ERROR] UDF migration: Error creating primary index on system collection for bucket standard_bucket0 - GSI CreatePrimaryIndex() - cause: ns_server _metakv returned: 503 Service Unavailable2024-01-22T21:17:01.978-08:00 [Fatal] MetakvGet: Failed to fetch /indexing/ddl/commandToken/delete/5848318864860926916 from metakv: ns_server _metakv returned: 503 Service Unavailable2024-01-22T21:17:01.978-08:00 [Error] Error in waitForScheduledIndex ns_server _metakv returned: 503 Service Unavailable

       

      Testrunner script to reproduce

      ./testrunner -i /data/workspace/debian-p0-ipv6-vset00-00-tools_1/testexec.27937.ini -p get-cbcollect-info=False,enable_ipv6=True,get-cbcollect-info=True,get-cbcollect-info=True,sirius_url=http://172.23.120.103:4000 -t clitest.docloadertest.docloaderTests.docloader_test,memory_quota=200,filename=gamesim-sample,standard_buckets=1,GROUP=P0

      Job name : debian-ipv6-tools_1

      Job ref : https://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-2032/jenkins_logs/test_suite_executor/668490/

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-60668 [Upgrade][Index] : Observing panic: HTTP error 503 Service Unavailable getting "http://127.0.0.1:8091/pools": "Temporary error occurred. Please try again later."

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MB-60686 Add retries to "503 Service Unavailable" error on`ns_server` API calls

          • Major - Major loss of function.
          • Open
          For Gerrit Dashboard: MB-60498
          # Subject Branch Project Status CR V
          204324,4 MB-60498 handle metakv unavailable in checkForTokens unstable indexing Status: NEW 0 0

          Activity

            People

              dhruvil.ketanshah Dhruvil Shah
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Gerrit Reviews

                  There is 1 open Gerrit change

                  PagerDuty