Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-58686

Auto failover timeout not honoured for stop_memcached scenario

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 7.6.0
    • 7.6.0
    • ns_server
    • 7.6.0-1507
      Centos 7 64bit

    Description

      Steps:

      • 5 node KV cluster with n2n encryption level=all
      • 1 magma bucket with replica=3
      • Set auto-failover timeout=1
      • Induce failure (stop_memcached using SIGSTOP) on '172.23.110.65'
      • Wait for auto-failover to happen

      Observations:

      From test POV, we are inducing the failures at "21:40:22.944" and the ns_server detects the node is down immediately,

      [ns_server:debug,2023-09-14T21:40:22.944-07:00,ns_1@172.23.110.64:<0.15702.0>:auto_failover:log_down_nodes_reason:403]Node 'ns_1@172.23.110.65' is considered down. Reason:"The data service did not respond. Either none of the buckets have warmed up or there is an issue with the data service. "

      But, the nserver is taking 5 seconds to trigger the auto-failover here,

      [ns_server:debug,2023-09-14T21:40:27.946-07:00,ns_1@172.23.110.64:<0.15700.0>:failover:start:44]Starting failover with Nodes = ['ns_1@172.23.110.65'], Options = #{allow_unsafe =>...

      TAF test:

       

      failover.concurrent_failovers.ConcurrentFailoverTests:
      		 
          test_concurrent_failover,nodes_init=5,services_init=kv-kv-kv-kv-kv,replicas=3,maxCount=1,timeout=1,failover_order=kv,failover_method=stop_memcached,bucket_spec=single_bucket.default

       

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-58636 Janitor cannot be cancelled if memcached is unresponsive in query_vbuckets_loop

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty