Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49134

cbbackupmgr restore failed on build 7.1.0-1558

    XMLWordPrintable

Details

    • Triaged
    • 1
    • Unknown
    • KV 2021-Oct-21, KV 2021-Nov

    Attachments

      Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          paolo.cocchi Paolo Cocchi added a comment - - edited

          Some collateral notes:

          1. The failure seen here is on builds that do NOT contain any bug introduced by MB-47318. Buggy change is https://github.com/couchbase/kv_engine/commit/bb20f27c9ba49fce2dadcc61b026b53c6227ef60 and it was introduced in 7.1.0-1571.
          2. I've repeated the test multiple times on the latest 7.1.0-1670, restore completes successfully. They're all the runs at http://perf.jenkins.couchbase.com/job/oceanus/ from 7367 to 7373.
          3. Some 7.1.0-1670 tests have run on a toy-build that includes stats enhancements by MB-48587. That has triggered MB-49469. To date, MB-49469 does NOT seem to cause the issue tracked here.
          4. This issue is probably a duplicate of MB-49037, where get the same high replica checkpoint mem-usage + ItemExpel not working as expected
          paolo.cocchi Paolo Cocchi added a comment - - edited Some collateral notes: The failure seen here is on builds that do NOT contain any bug introduced by MB-47318 . Buggy change is https://github.com/couchbase/kv_engine/commit/bb20f27c9ba49fce2dadcc61b026b53c6227ef60 and it was introduced in 7.1.0-1571. I've repeated the test multiple times on the latest 7.1.0-1670, restore completes successfully. They're all the runs at http://perf.jenkins.couchbase.com/job/oceanus/ from 7367 to 7373. Some 7.1.0-1670 tests have run on a toy-build that includes stats enhancements by MB-48587 . That has triggered MB-49469 . To date, MB-49469 does NOT seem to cause the issue tracked here. This issue is probably a duplicate of MB-49037 , where get the same high replica checkpoint mem-usage + ItemExpel not working as expected
          paolo.cocchi Paolo Cocchi added a comment -

          As mentioned in the previous update, MB-49037 seems a duplicate of this MB. The issue more reproducible on MB-49037 than here, so I'm aiming to run a live-debugging session there. Hopefully that will tell us what's going on at ItemExpel.

          paolo.cocchi Paolo Cocchi added a comment - As mentioned in the previous update, MB-49037 seems a duplicate of this MB. The issue more reproducible on MB-49037 than here, so I'm aiming to run a live-debugging session there. Hopefully that will tell us what's going on at ItemExpel.
          paolo.cocchi Paolo Cocchi added a comment - - edited

          Further investigation in MB-49037 reveals that we are seeing a HashTable ejection issue there. See https://issues.couchbase.com/secure/EditComment!default.jspa?id=174299&commentId=560681. So this isn't expected to be a duplicate of MB-49037 at this point.

          paolo.cocchi Paolo Cocchi added a comment - - edited Further investigation in MB-49037 reveals that we are seeing a HashTable ejection issue there. See https://issues.couchbase.com/secure/EditComment!default.jspa?id=174299&commentId=560681 . So this isn't expected to be a duplicate of MB-49037 at this point.
          paolo.cocchi Paolo Cocchi added a comment - - edited

          Hi Bo-Chun Wang,
          I'm resolving this as "Cannot Reproduce".
          Summary is that the failure seen on build 1554 shows symptoms on stats that suggest some issue at ItemExpel. But:

          • I've never managed to reproduce it on your env, and no repro on local envs either
          • In 1554 there are stats inconsistency that make checkpoint (memory) stats not totally reliable

          Newer Neo build will shortly contain all the necessary stats improvements (MB-48587). Please re-open this ticket if you hit the issue again.

          Thank you

          paolo.cocchi Paolo Cocchi added a comment - - edited Hi Bo-Chun Wang , I'm resolving this as "Cannot Reproduce". Summary is that the failure seen on build 1554 shows symptoms on stats that suggest some issue at ItemExpel. But: I've never managed to reproduce it on your env, and no repro on local envs either In 1554 there are stats inconsistency that make checkpoint (memory) stats not totally reliable Newer Neo build will shortly contain all the necessary stats improvements ( MB-48587 ). Please re-open this ticket if you hit the issue again. Thank you

          Closing since this is non-reproducible

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Closing since this is non-reproducible

          People

            bo-chun.wang Bo-Chun Wang
            bo-chun.wang Bo-Chun Wang
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty