Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48314

Persist rollbacks stats in case of indexer crash - needed for verification

    XMLWordPrintable

Details

    • 1

    Description

      Killing an indexer or indexer crash resets all the rollback stats. We need to persists these stats for QE validation as in certain scenario we do comparisons between stats value before and after indexer kill/crash for recovery.

       

      hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
        "MAINT_STREAM:default:num_rollbacks": 2,
        "MAINT_STREAM:default:num_rollbacks_to_zero": 1,
      hemantrajput@LFC testrunner % ssh root@10.112.205.101                                                                                     
      Last login: Fri Sep  3 07:29:24 2021 from 10.112.205.1
      [root@node1-cb660-centos7 ~]# pgrep indexer
      25923
      [root@node1-cb660-centos7 ~]# kill -9 25923
      [root@node1-cb660-centos7 ~]# exit
      logout
      Connection to 10.112.205.101 closed.
      hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
        "MAINT_STREAM:default:num_rollbacks": 0,
        "MAINT_STREAM:default:num_rollbacks_to_zero": 0,
      hemantrajput@LFC testrunner % 

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-48314
          # Subject Branch Project Status CR V

          Activity

            hemant.rajput Hemant Rajput created issue -
            jeelan.poola Jeelan Poola made changes -
            Field Original Value New Value
            Fix Version/s Neo [ 17615 ]
            Fix Version/s 7.0.2 [ 18012 ]
            jeelan.poola Jeelan Poola made changes -
            Issue Type Improvement [ 4 ] Bug [ 1 ]
            jeelan.poola Jeelan Poola made changes -
            Summary Persist rollbacks stats in case of indexer crash Persist rollbacks stats in case of indexer crash - needed for verification
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Varun Velamuri [ varun.velamuri ]
            jeelan.poola Jeelan Poola made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            varun.velamuri Varun Velamuri added a comment - - edited

            Hemant Rajput, Can you please elaborate your requirements around the persistence? Currently, we have two types of persistences (a) Flat file (b) Snapshot. Flat file persistence happens every 15 min. This interval is configurable. Snapshotting happens every 10 minutes and it is per index. I don't think it is a good idea to change snapshotting interval in recovery tests.

            So, I am thinking to persist the stat in a file on disk and read it back when recovering. If you need more fine grained interval, change the  "indexer.statsPersistenceInterval" to a less value (currently at 900). I believe this will give you enough control over the stat. Please let me know if you have any other requirement.

            varun.velamuri Varun Velamuri added a comment - - edited Hemant Rajput , Can you please elaborate your requirements around the persistence? Currently, we have two types of persistences (a) Flat file (b) Snapshot. Flat file persistence happens every 15 min. This interval is configurable. Snapshotting happens every 10 minutes and it is per index. I don't think it is a good idea to change snapshotting interval in recovery tests. So, I am thinking to persist the stat in a file on disk and read it back when recovering. If you need more fine grained interval, change the  "indexer.statsPersistenceInterval" to a less value (currently at 900). I believe this will give you enough control over the stat. Please let me know if you have any other requirement.

            Varun Velamuri, my requirement is pretty simple. I just want to retrieve if the rollback happened to zero after a certain action. As, "num_rollback_to_zero"  is a cumulative stats we need to persist it as we do comparison of the value of stats before and after certain action.

            Hope this makes it lucid. Let me know if you need any more information from me.

            hemant.rajput Hemant Rajput added a comment - Varun Velamuri , my requirement is pretty simple. I just want to retrieve if the rollback happened to zero after a certain action. As, "num_rollback_to_zero"  is a cumulative stats we need to persist it as we do comparison of the value of stats before and after certain action. Hope this makes it lucid. Let me know if you need any more information from me.
            jeelan.poola Jeelan Poola made changes -
            Link This issue is cloned by MB-48336 [ MB-48336 ]
            jeelan.poola Jeelan Poola made changes -
            Fix Version/s 7.0.2 [ 18012 ]

            Build couchbase-server-7.1.0-1260 contains indexing commit f42a632 with commit message:
            MB-48314 Persist num_rollback and num_rollbacks_to_zero stats

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1260 contains indexing commit f42a632 with commit message: MB-48314 Persist num_rollback and num_rollbacks_to_zero stats

            Varun Velamuri Can this be resolved now?

            jeelan.poola Jeelan Poola added a comment - Varun Velamuri Can this be resolved now?
            varun.velamuri Varun Velamuri made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Hemant Rajput [ hemant.rajput ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Done

            varun.velamuri Varun Velamuri added a comment - Done

            Validated on 7.10-1695

             

            Stats are being persisted every 15min. However, I changed, the interval to 15s and tried it.

             

             

            curl -u Administrator:password -X POST  http://10.112.205.101:9102/settings -d '{"indexer.statsPersistenceInterval": 15}'

             

            hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
              "MAINT_STREAM:travel-sample:num_rollbacks": 0,
              "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 0,
             
            Flush the bucket
            sleep(15s)
            pkill indexer
             
            hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
              "MAINT_STREAM:travel-sample:num_rollbacks": 1,
              "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 1,
             
            Flush the bucket
            sleep(15s)
            pkill indexer
             
            hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
              "MAINT_STREAM:travel-sample:num_rollbacks": 2,
              "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 2,
            hemant.rajput Hemant Rajput added a comment - Validated on 7.10-1695   Stats are being persisted every 15min. However, I changed, the interval to 15s and tried it.     curl -u Administrator:password -X POST  http://10.112.205.101:9102/settings -d '{"indexer.statsPersistenceInterval": 15}'   hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks    "MAINT_STREAM:travel-sample:num_rollbacks": 0,   "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 0,   Flush the bucket sleep(15s) pkill indexer   hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks    "MAINT_STREAM:travel-sample:num_rollbacks": 1,   "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 1,   Flush the bucket sleep(15s) pkill indexer   hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks    "MAINT_STREAM:travel-sample:num_rollbacks": 2,   "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 2,
            hemant.rajput Hemant Rajput made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              hemant.rajput Hemant Rajput
              hemant.rajput Hemant Rajput
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty