Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61440

[System Test] cluster stuck in balanced state even after multiple nodes failover/addback/addition

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • 7.2.6
    • 7.2.5
    • ns_server
    • Enterprise Edition 7.2.5 build 7587
    • Untriaged
    • 0
    • Unknown

    Description

      During this system test cluster reached a bad state when multiple data nodes reached disk full state which caused them to be in down state
      I tried to recover one of them manually **
      by failing  over 172.23.106.109 which succeeded ,
      then deleting the 'default' bucket which succeeded as well .
      and trying to add back 172.23.97.229 to cluster but realised the cluster was already in balanced state even with all of the above conditions

      logs from .224

       {view_fragmentation_threshold,{30,undefined}}]
      [user:info,2024-04-07T12:03:09.779-07:00,ns_1@172.23.97.244:ns_memcached-default<0.15594.4>:ns_memcached:do_terminate:855]Shutting down bucket "default" on 'ns_1@172.23.97.244' for deletion
      [ns_server:info,2024-04-07T12:03:09.779-07:00,ns_1@172.23.97.244:ns_memcached-default<0.15594.4>:ns_memcached:delete_bucket:884]Deleting bucket "default" from memcached (force = true)
      [ns_server:info,2024-04-07T12:03:10.132-07:00,ns_1@172.23.97.244:<0.5819.1026>:compaction_daemon:spawn_scheduled_views_compactor:548]Start compaction of indexes for bucket ITEM with config: 
      [{parallel_db_and_view_compaction,false},
       {database_fragmentation_threshold,{30,undefined}},
       {view_fragmentation_threshold,{30,undefined}}]
      [ns_server:info,2024-04-07T12:03:10.135-07:00,ns_1@172.23.97.244:<0.6685.1026>:compaction_daemon:spawn_scheduled_views_compactor:548]Start compaction of indexes for bucket NEW_ORDER with config: 
      [{parallel_db_and_view_compaction,false},
       {database_fragmentation_threshold,{30,undefined}},
       {view_fragmentation_threshold,{30,undefined}}]
      [ns_server:info,2024-04-07T12:03:10.138-07:00,ns_1@172.23.97.244:<0.5012.1026>:compaction_daemon:spawn_scheduled_views_compactor:548]Start compaction of indexes for bucket WAREHOUSE with config: 
      [{parallel_db_and_view_compaction,false},
       {database_fragmentation_threshold,{30,undefined}},
       {view_fragmentation_threshold,{30,undefined}}]
      [ns_server:info,2024-04-07T12:03:10.142-07:00,ns_1@172.23.97.244:

      from http://172.23.97.244:8091/pools/default

      just_validate=1"}},"rebalanceStatus":"none","rebalanceProgressUri":"/pools/default/rebalanceProgress","stopRebalanceUri":"/controller/stopRebalance?uuid=441de1be2aacab9d31d169b37a2292a5","nodeStatusesUri":"/nodeStatuses","nodeServicesUri":"/pools/default/nodeServices?v=112172401","maxBucketCount":30,"maxCollectionCount":1200,"maxScopeCount":1200,"autoCompactionSettings":{"parallelDBAndViewCompaction":false,"magmaFragmentationPercentage":50,"databaseFragmentationThreshold":{"percentage":30,"size":"undefined"},"viewFragmentationThreshold":{"percentage":30,"size":"undefined"},"indexCompactionMode":"circular","indexCircularCompaction":{"daysOfWeek":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","interval":
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Pramada.Kolichala Pramada Kolichala
            pulkit.matta Pulkit Matta
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty