Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53201

[6.6.5-10106] [Windows] - Secondary Index rollback to zero after KV node auto failover

    XMLWordPrintable

Details

    • Untriaged
    • Windows 64-bit
    • 1
    • Unknown

    Description

      Wanted to give a shot at repro'ing MB-53057 on windows. In the past we have seen memory/swap usage being extremely high in windows compared to linux for the exact same test and configuration.

      Steps to Repro
      1. Create a 6 node cluster with 3kv, 2 indexing and 1 n1ql nodes.
      2. Create buckets/data/indexes and push buckets to dgm and ensure indexes are in DGM as well. Start running queries in background with request_plus consistency level.
      3. Kill memcached on one of the kv nodes(172.23.138.72). Node gets autofailed over.

      $ tasklist | grep memcached
      memcached.exe                 4288 Services                   0  1,845,236 K
       
      $ taskkill /F /T /IM memcached*
       
       
      SUCCESS: The process with PID 4288 (child process of PID 5064) has been terminated.
       
      Administrator@WIN-2UALDR3QV3R ~
      $ tasklist | grep memcached
      memcached.exe                 5096 Services                   0    250,612 K
       
      Administrator@WIN-2UALDR3QV3R ~
      $ 
      

      Do a full recovery and rebalance the node back in. 

      Saw rollbackAllToZero messages on both the indexer nodes.

      172.23.138.77

      Administrator@WIN-2UALDR3QV3R ~
      $ cd /cygdrive/c/Program\ Files/Couchbase/Server/var/lib/couchbase/logs/
       
      Administrator@WIN-2UALDR3QV3R /cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/logs
      $ grep rollbackAllToZero indexer.log 
      2022-08-01T00:58:14.528-07:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM test1
      2022-08-01T00:59:15.709-07:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM test1
       
      Administrator@WIN-2UALDR3QV3R /cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/logs
      $ 
      

      172.23.138.79

      Administrator@WIN-2UALDR3QV3R ~
      $ cd /cygdrive/c/Program\ Files/Couchbase/Server/var/lib/couchbase/logs/
       
      Administrator@WIN-2UALDR3QV3R /cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/logs
      $ grep rollbackAllToZero indexer.log 
      2022-08-01T00:58:13.191-07:00 [Info] StorageMgr::rollbackAllToZero MAINT_STREAM test1
       
      Administrator@WIN-2UALDR3QV3R /cygdrive/c/Program Files/Couchbase/Server/var/lib/couchbase/logs
      $
      

      From the UI Indexes page :-

      cbcollect_info attached.

      Attachments

        Activity

          People

            varun.velamuri Varun Velamuri
            Balakumaran.Gopal Balakumaran Gopal
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty