Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47267

Vbucket stats call to KV can timeout during delta node recovery preparation

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • KV-Engine Sprint 2021 July, KV-Engine Sprint 2021 August

    Description

      During delta node recovery, ns_server needs to be able to access vbucket state information during bucket initialization (i.e. before the bucket is fully warmed). This information is needed to prepare the vbuckets for delta node recovery (i.e. doing such things as maybe setting vbucket to replica state, deleting vbuckets with incompatible vbucket histories, etc.)

      There's a 180s timeout to get the vbucket state information, but recently in the field we've seen a case where it can take longer than that for this information to be returned as KV doesn't return stats information on vbuckets until "vbucket maps are populated" and this can take more than 180s.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-1236 contains kv_engine commit cf3d145 with commit message:
            MB-47267: Simplify WarmupBackfillTask::finishTask()

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1236 contains kv_engine commit cf3d145 with commit message: MB-47267 : Simplify WarmupBackfillTask::finishTask()

            Build couchbase-server-7.1.0-1236 contains kv_engine commit 60e1f16 with commit message:
            MB-47267: Use RO KVStore for Warmup

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1236 contains kv_engine commit 60e1f16 with commit message: MB-47267 : Use RO KVStore for Warmup

            Build couchbase-server-7.1.0-1236 contains kv_engine commit 423ad54 with commit message:
            Recommit: MB-47267: Make backfill during warmup a PauseResume task

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1236 contains kv_engine commit 423ad54 with commit message: Recommit: MB-47267 : Make backfill during warmup a PauseResume task

            Meni Hillel Dave Finlay requesting the dev team to close this bug based on dev testing.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Meni Hillel Dave Finlay requesting the dev team to close this bug based on dev testing.
            richard.demellow Richard deMellow added a comment - - edited

            Hi Mihir Kamdar, I'll close this ticket as we do have a regression test within our KV unit tests. Though if QE wanted to add test for this the following steps would verify this:

            1. Create a single KV node
            2. Create two buckets A and B
            3. Add a large number of documents to all vbucket in bucket A
            4. Add few documents to each vbucket in bucket B (orders of magnitude less)
            5. Shutdown and warmup the node (with node A warming up first)
            6. Verify that we're able to access vbucket state of each vbucket for bucket B before bucket A is fully warmed up.
              1. With the idea that the warmup of bucket B isn't blocked by the warmup for bucket A despite bucket A having a large number of documents.
            richard.demellow Richard deMellow added a comment - - edited Hi Mihir Kamdar , I'll close this ticket as we do have a regression test within our KV unit tests. Though if QE wanted to add test for this the following steps would verify this: Create a single KV node Create two buckets A and B Add a large number of documents to all vbucket in bucket A Add few documents to each vbucket in bucket B (orders of magnitude less) Shutdown and warmup the node (with node A warming up first) Verify that we're able to access vbucket state of each vbucket for bucket B before bucket A is fully warmed up. With the idea that the warmup of bucket B isn't blocked by the warmup for bucket A despite bucket A having a large number of documents.

            People

              dfinlay Dave Finlay
              dfinlay Dave Finlay
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty