Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48336

[BP 7.0.2 MB-48314] - Persist rollbacks stats in case of indexer crash - needed for verification

    XMLWordPrintable

Details

    • 1

    Description

      Killing an indexer or indexer crash resets all the rollback stats. We need to persists these stats for QE validation as in certain scenario we do comparisons between stats value before and after indexer kill/crash for recovery.

       

      hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
      		 
        "MAINT_STREAM:default:num_rollbacks": 2,
      		 
        "MAINT_STREAM:default:num_rollbacks_to_zero": 1,
      		 
      hemantrajput@LFC testrunner % ssh root@10.112.205.101                                                                                     
      		 
      Last login: Fri Sep  3 07:29:24 2021 from 10.112.205.1
      		 
      [root@node1-cb660-centos7 ~]# pgrep indexer
      		 
      25923
      		 
      [root@node1-cb660-centos7 ~]# kill -9 25923
      		 
      [root@node1-cb660-centos7 ~]# exit
      		 
      logout
      		 
      Connection to 10.112.205.101 closed.
      		 
      hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
      		 
        "MAINT_STREAM:default:num_rollbacks": 0,
      		 
        "MAINT_STREAM:default:num_rollbacks_to_zero": 0,
      		 
      hemantrajput@LFC testrunner % 

      Attachments

        Issue Links

          Clones

          Bug - A problem which impairs or prevents the functions of the product. MB-48314 Persist rollbacks stats in case of indexer crash - needed for verification

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            jeelan.poola Jeelan Poola created issue -
            jeelan.poola Jeelan Poola made changes -
            Field Original Value New Value
            Link This issue Clones MB-48314 [ MB-48314 ]
            jeelan.poola Jeelan Poola made changes -
            Fix Version/s Neo [ 17615 ]
            jeelan.poola Jeelan Poola made changes -
            Link This issue blocks MB-46308 [ MB-46308 ]
            wayne Wayne Siu made changes -
            Labels functional-test approved-for-7.0.2 functional-test
            varun.velamuri Varun Velamuri made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.2-6654 contains indexing commit d5ef38c with commit message:
            MB-48336 Persist num_rollback and num_rollbacks_to_zero stats

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6654 contains indexing commit d5ef38c with commit message: MB-48336 Persist num_rollback and num_rollbacks_to_zero stats
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Hemant Rajput [ hemant.rajput ]
            hemant.rajput Hemant Rajput added a comment -

            Validated on 7.0.2-6671

             

            Stats are being persisted every 15min. However, I changed, the interval to 15s and tried it.

             

            curl -u Administrator:password -X POST  http://10.112.205.101:9102/settings -d '{"indexer.statsPersistenceInterval": 15}'
            		 

            hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
            		 
              "MAINT_STREAM:travel-sample:num_rollbacks": 0,
            		 
              "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 0,
            		 
             
            		 
            Flush the bucket
            		 
            sleep(15s)
            		 
            pkill indexer
            		 
             
            		 
            hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
            		 
              "MAINT_STREAM:travel-sample:num_rollbacks": 1,
            		 
              "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 1,
            		 
             
            		 
            Flush the bucket
            		 
            sleep(15s)
            		 
            pkill indexer
            		 
             
            		 
            hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks 
            		 
              "MAINT_STREAM:travel-sample:num_rollbacks": 2,
            		 
              "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 2,

            hemant.rajput Hemant Rajput added a comment - Validated on 7.0.2-6671   Stats are being persisted every 15min. However, I changed, the interval to 15s and tried it.   curl -u Administrator:password -X POST  http://10.112.205.101:9102/settings -d '{"indexer.statsPersistenceInterval": 15}' hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks    "MAINT_STREAM:travel-sample:num_rollbacks": 0,   "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 0,   Flush the bucket sleep(15s) pkill indexer   hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks    "MAINT_STREAM:travel-sample:num_rollbacks": 1,   "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 1,   Flush the bucket sleep(15s) pkill indexer   hemantrajput@LFC testrunner % curl -s -v -u Administrator:password http://10.112.205.101:9102/stats 2>/dev/null | jq | grep num_rollbacks    "MAINT_STREAM:travel-sample:num_rollbacks": 2,   "MAINT_STREAM:travel-sample:num_rollbacks_to_zero": 2,
            hemant.rajput Hemant Rajput made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              hemant.rajput Hemant Rajput
              jeelan.poola Jeelan Poola
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty