Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-36468

[Volume]: Data mismatch for Timers and SBM

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      Build: 6.5.0-4558

      Test: MH eventing volume

      Seeing data mismatch for eventing 

      Source bucket : 41,463,650

      Timers: 41,463,555 (missing = 95)

      SBM: 82,927,258 (41,463,650*2 = 82,927,300 missing : 42)

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-1093 contains eventing commit 8fb1b10 with commit message:
            MB-36468: Fix for data mismatch in timers

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1093 contains eventing commit 8fb1b10 with commit message: MB-36468 : Fix for data mismatch in timers

            Build couchbase-server-7.0.0-1093 contains eventing-ee commit 81d57b4 with commit message:
            MB-36468 Merge remote-tracking branch 'couchbase-priv/unstable' into HEAD

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1093 contains eventing-ee commit 81d57b4 with commit message: MB-36468 Merge remote-tracking branch 'couchbase-priv/unstable' into HEAD

            Build couchbase-server-7.0.0-1093 contains eventing-ee commit fdd0984 with commit message:
            MB-36468: Fix for data mismatch in timers

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1093 contains eventing-ee commit fdd0984 with commit message: MB-36468 : Fix for data mismatch in timers
            jeelan.poola Jeelan Poola added a comment -

            SBM failures are likely due to low resident ratio (5.75%) on sbm_op_dst bucket. We see a lot of TMP_FAIL errors like below

            2019-12-02T02:09:04.637-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.637-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.633-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.632-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.631-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.631-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.630-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            2019-12-02T02:09:04.630-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"}
            

            All timers related fixes are merged and eventing repos are branched out. Requesting Vikas Chaudhary to validate SBM failures with a higher resident ratio (>30%) on sbm_op_dst bucket.

            jeelan.poola Jeelan Poola added a comment - SBM failures are likely due to low resident ratio (5.75%) on sbm_op_dst bucket. We see a lot of TMP_FAIL errors like below 2019-12-02T02:09:04.637-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.637-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.633-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.632-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.631-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.631-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.630-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} 2019-12-02T02:09:04.630-08:00 [INFO] {"message":{"code":392,"desc":"Temporary failure received from server. Try again later","name":"LCB_ETMPFAIL"},"stack":"Error\n at OnUpdate (bucket_op_sbm.js:4:31)"} All timers related fixes are merged and eventing repos are branched out. Requesting Vikas Chaudhary to validate SBM failures with a higher resident ratio (>30%) on sbm_op_dst bucket.

            volume passed on 6.5.0-4913

            vikas.chaudhary Vikas Chaudhary added a comment - volume passed on 6.5.0-4913

            People

              satya.nand Satya Nand (Inactive)
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty