Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45989

FTS: rollback not happening - FTS not catching up with KV

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0-5017
      Test Suite: moving-topology-scorch_5.5_P1
      Rollback Tests:

      ./testrunner -i /tmp/testexec.21146.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P1,index_type=scorch,fts_quota=990 -t fts.moving_topology_fts.MovingTopFTS.partial_rollback,cluster=D,D,F,items=50000,update=True,upd=30,upd_del_fields=['dept'],GROUP=P1,get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,index_type=scorch,fts_quota=990

      ./testrunner -i /tmp/testexec.21146.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P1,index_type=scorch,fts_quota=990 -t fts.moving_topology_fts.MovingTopFTS.partial_rollback,cluster=D,D,F,items=50000,delete=True,del=10,GROUP=P1,get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,index_type=scorch,fts_quota=990

      Test steps:

      • Cluster setup : n1:kv, n2:kv, n3:search
      • Load 50000 with field and value : mutated:0
      • Create default index
      • Stop persistence on both kv nodes
      • Update 15000 docs with mutated:1
      • Get the query count before rollback: {"query": "mutated:>0"}

        which is 15000

      • Kill memcached on n1
      • start persistence on n2
      • Failover n2
      • Kv rolled back and updated 15000 docs reverted back to before update. So they reverted back to mutated:0
      • We expect FTS index also should rollback and update the index along with kv. But its not.
      • Query {"query": "mutated:>0"}

        returns 15000, but in KV there are 0 docs

      curl -XGET  http://172.23.96.141:8094/api/nsstats -u Administrator:password | jq | grep rollback
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
      100  3594    0  3594    0     0  20078      0 --:--:-- --:--:-- --:--:-- 20191
        "default:default_index:timer_rollback_count": 0,
        "tot_rollback_full": 0,
        "tot_rollback_partial": 0,
      

      Attached screenshot where mutated:1 returns 15000 results, but actually looking into the doc, mutated is 0

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rollback/collectinfo-2021-04-29T003740-ns_1%40172.23.96.141.zip [FTS]
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rollback/collectinfo-2021-04-29T003740-ns_1%40172.23.97.211.zip [KV]
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/rollback/collectinfo-2021-04-29T003740-ns_1%40172.23.97.212.zip [KV]

      Attachments

        1. fts.log
          5.01 MB
          Abhi Dangeti
        2. Screen Shot 2021-04-28 at 5.17.52 PM.png
          144 kB
          Girish Benakappa

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              abhinav Abhi Dangeti
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty