Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-31322

[CX] Subsequent Rebalance Fails After Cancellation With Secondary Indexes

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 6.0.0
    • 6.0.0
    • analytics
    • Untriaged
    • Unknown
    • CX Sprint 120

    Description

      If a rebalance is canceled during certain stages of rebalancing secondary indexes, any subsequent rebalance will fail with the exception:

      HYR0081: File (file_path) is already mapped
      

      This happens due to improper cleanup from the previously cancelled rebalance attempt.

      Note that in other stages of the rebalance, the cancellation might cause AnalyticsDriver process to restart to ensure data integrity. When that happens, the exception will not be encountered.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Murtadha Hubail, we can have secondary indexes created and then cancel rebalance out/in. Any sure-shot way to validate this? 

            tanzeem.ahmed Tanzeem Ahmed (Inactive) added a comment - Murtadha Hubail , we can have secondary indexes created and then cancel rebalance out/in. Any sure-shot way to validate this? 
            murtadha.hubail Murtadha Hubail added a comment - - edited

            Tanzeem Ahmed,

            This has to do with the time the rebalance is cancelled. To increase the chance of it happening, you can create multiple secondary indexes and cancel the rebalance at random times. The subsequent rebalance shouldn't fail after the cancellation. As I said, there is a chance that the cancellation will cause a restart to the java process, but that restart is by design.

            murtadha.hubail Murtadha Hubail added a comment - - edited Tanzeem Ahmed , This has to do with the time the rebalance is cancelled. To increase the chance of it happening, you can create multiple secondary indexes and cancel the rebalance at random times. The subsequent rebalance shouldn't fail after the cancellation. As I said, there is a chance that the cancellation will cause a restart to the java process, but that restart is by design.

            Verified with rebalance in and out.

            1. Created dataset and 4 secondary index on a KV bucket with 200k documents
            2. Cancelled rebalance(In/Out) at different rebalance percentages(4.5%, 12.8%, 20.4%, 33%). Beyond this rebalance just completes on CBAS nodes
            3. Verified cluster was stable by running select queries
            4. Verified subsequent rebalance passes
            tanzeem.ahmed Tanzeem Ahmed (Inactive) added a comment - Verified with rebalance in and out. Created dataset and 4 secondary index on a KV bucket with 200k documents Cancelled rebalance(In/Out) at different rebalance percentages(4.5%, 12.8%, 20.4%, 33%). Beyond this rebalance just completes on CBAS nodes Verified cluster was stable by running select queries Verified subsequent rebalance passes

            People

              tanzeem.ahmed Tanzeem Ahmed (Inactive)
              murtadha.hubail Murtadha Hubail
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty