Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7717

memcached ignores ns_server request(s) to deregister replica building tap names after adding back failed-over node

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.1
    • Fix Version/s: 3.0
    • Component/s: couchbase-bucket
    • Security Level: Public
    • Labels:
      None
    • Triage:
      Untriaged

      Description

      Please see details explanation in MB-5906.

      Started over with a pair of single node clusters (10.1.3.71 and 10.1.3.72), threads=4, expiration=10
      – Added a node (10.1.3.235) on to the source cluster, rebalance.

      • Fall in destination ops-per-sec towards the end of rebalance on the source`s side.
      • But picks up once rebalance actually finishes.
        • Failover a node on source cluster
      • Couldn`t add it back and rebalance for some reason, so removed it completely, rebalanced and then added another node again, but rebalance is now stuck (between 10.1.3.71 and 10.1.3.235)

      Basically, memcached timed out on numerous request(s) to deregister replica building tap from ns_server. Thus post-rebalance clean up indefinitely got stuck.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        mikew Mike Wiederhold added a comment -

        We may need to try to reproduce this issue. I will follow up later, but this will likely get pushed to 2.1.

        Show
        mikew Mike Wiederhold added a comment - We may need to try to reproduce this issue. I will follow up later, but this will likely get pushed to 2.1.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        deferring out of 2.0.2

        Show
        maria Maria McDuff (Inactive) added a comment - deferring out of 2.0.2
        Hide
        maria Maria McDuff (Inactive) added a comment -

        sangharsh,

        can you pls try to repro in 2.2.0 build 821?
        pls update this bug with your findings.
        base on your test result, we will decide if this bug is a must fix for 2.5.
        thanks.

        Show
        maria Maria McDuff (Inactive) added a comment - sangharsh, can you pls try to repro in 2.2.0 build 821? pls update this bug with your findings. base on your test result, we will decide if this bug is a must fix for 2.5. thanks.
        Hide
        maria Maria McDuff (Inactive) added a comment -

        Sangharsh,

        is this still happening in 2.5.0?
        pls close if it's no longer an issue in 2.5.0.

        Show
        maria Maria McDuff (Inactive) added a comment - Sangharsh, is this still happening in 2.5.0? pls close if it's no longer an issue in 2.5.0.
        Hide
        iryna iryna added a comment -

        didn't reproduce it for build 2.5.0-1058
        steps scenario 1:
        – Added a node on to the source cluster, rebalance under load.
        – Failover a node on source cluster -> add it back
        steps scenario 2:
        – Added a node on to the source cluster, rebalance under load.
        – Failover a node on source cluster -> rebalance
        – add another node and rebalance

        Show
        iryna iryna added a comment - didn't reproduce it for build 2.5.0-1058 steps scenario 1: – Added a node on to the source cluster, rebalance under load. – Failover a node on source cluster -> add it back steps scenario 2: – Added a node on to the source cluster, rebalance under load. – Failover a node on source cluster -> rebalance – add another node and rebalance

          People

          • Assignee:
            parag Parag Agarwal (Inactive)
            Reporter:
            jin Jin Lim (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes