Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50034

Potential deadlock in MOI doPersistSnapshot 6.6.3

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes

    Description

      MOI doPersistSnapshot is missing mutex unlock, this can cause deadlock and cause Indexer main loop to get stuck, which will lead to multiple problems including but not limited to 

      rebalance failure

      stuck flush threads etc etc.

      This is regression introduced with  https://issues.couchbase.com/browse/MB-46507.

      This is a regression that was introduced in 6.6.3.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-6.6.5-10064 contains indexing commit 1eea12d with commit message:
            MB-50034: Potential deadlock in MOI.doPersistSnapshot

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.5-10064 contains indexing commit 1eea12d with commit message: MB-50034 : Potential deadlock in MOI.doPersistSnapshot

            Was able to reproduce the problem easily by running the MOI system test (and doing some indexer setting changes while the test was running) with build 6.6.5-10063. Could see multiple rebalance failures in a 5 hr run.

            WIth the fix, with the same test with build 6.6.5-10071, the issue was not seen in a test run spanning 40+ hrs.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Was able to reproduce the problem easily by running the MOI system test (and doing some indexer setting changes while the test was running) with build 6.6.5-10063. Could see multiple rebalance failures in a 5 hr run. WIth the fix, with the same test with build 6.6.5-10071, the issue was not seen in a test run spanning 40+ hrs.

            Hemant Rajput we need to automate this. Steps:

            1) create MOI index
            2) do some mutations and wait for persistent snapshot to be created
            3) change some config/settings
            4) do some more mutations
            5) now check if new mutations are being indexed/try some DDL/or rebalance - these should work seamlessly.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Hemant Rajput we need to automate this. Steps: 1) create MOI index 2) do some mutations and wait for persistent snapshot to be created 3) change some config/settings 4) do some more mutations 5) now check if new mutations are being indexed/try some DDL/or rebalance - these should work seamlessly.

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              yogendra.acharya Yogendra Acharya (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty