Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-55462

[6.6.6-MP1][System Test] Failover exited with reason config_sync_failed.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • None
    • 6.6.6
    • ns_server
    • Enterprise Edition 6.6.6 build 10579

    Description

      QE TEST

      -test	tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml

      Day - 1
      Cycle - 1
      Scale - 3

      Sequence of events

      KV node 172.23.105.0 went down due to disk full issue.

      2023-02-06T19:41:14.817-08:00, compaction_daemon:0:critical:message(ns_1@172.23.105.0) - Cannot compact database `default/99`: the estimated necessary disk space is about 17478426 bytes but the currently available disk space is 0 bytes. (repeated 1 times, last seen 66.027874 secs ago)
      		 
      2023-02-06T19:41:14.817-08:00, compaction_daemon:0:critical:message(ns_1@172.23.105.0) - Cannot compact view `mapreduce_view/default/_design/scale/main`: the estimated necessary disk space is about 594966628 bytes but the currently available disk space is 0 bytes. (repeated 1 times, last seen 67.184842 secs ago)
      		 
      2023-02-06T19:41:14.817-08:00, ns_memcached:0:info:message(ns_1@172.23.105.0) - Control connection to memcached on 'ns_1@172.23.105.0' disconnected. Check logs for details. (repeated 9 times, last seen 51.479492 secs ago)

      Auto Failover of this node was triggered as expected.

      2023-02-06T19:47:50.228-08:00, failover:0:info:message(ns_1@172.23.104.176) - Starting failing over ['ns_1@172.23.105.0']
      		 
      2023-02-06T19:47:50.228-08:00, ns_orchestrator:0:info:message(ns_1@172.23.104.176) - Starting failover of nodes ['ns_1@172.23.105.0']. Operation Id = 35bd3b6140de206c8b95172f6965bee9

      Failover operation failed with following error.

      [ns_server:error,2023-02-06T19:48:01.490-08:00,ns_1@172.23.104.176:<0.30980.170>:failover:pre_failover_config_sync:138]Config pull from ['ns_1@172.23.104.176','ns_1@172.23.104.249',
      		 
                        'ns_1@172.23.105.134','ns_1@172.23.105.210',
      		 
                        'ns_1@172.23.105.38','ns_1@172.23.105.39',
      		 
                        'ns_1@172.23.105.91','ns_1@172.23.106.32',
      		 
                        'ns_1@172.23.106.37','ns_1@172.23.106.54',
      		 
                        'ns_1@172.23.107.142','ns_1@172.23.107.236',
      		 
                        'ns_1@172.23.107.25','ns_1@172.23.108.129',
      		 
                        'ns_1@172.23.108.132','ns_1@172.23.108.134',
      		 
                        'ns_1@172.23.108.136','ns_1@172.23.108.138',
      		 
                        'ns_1@172.23.108.139','ns_1@172.23.108.140',
      		 
                        'ns_1@172.23.108.141','ns_1@172.23.108.143',
      		 
                        'ns_1@172.23.108.144','ns_1@172.23.108.145',
      		 
                        'ns_1@172.23.108.146','ns_1@172.23.108.148',
      		 
                        'ns_1@172.23.108.34','ns_1@172.23.108.61'] failed: {error,
      		 
                                                                            {get_compressed_failed,
      		 
                                                                             [{'ns_1@172.23.107.142',
      		 
                                                                               timeout}]}}
      		 
      [user:error,2023-02-06T19:48:01.502-08:00,ns_1@172.23.104.176:<0.30980.170>:failover:orchestrate:99]Failover failed with config_sync_failed
      		 
      [user:error,2023-02-06T19:48:01.504-08:00,ns_1@172.23.104.176:<0.6994.170>:ns_orchestrator:log_rebalance_completion:1460]Failover exited with reason config_sync_failed.
      		 
      Rebalance Operation Id = 35bd3b6140de206c8b95172f6965bee9

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-55431 We should make all updates to bucket config during failover in a single transaction

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              navdeep.boparai Navdeep Boparai
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty