Uploaded image for project: 'Couchbase Java Client'
  1. Couchbase Java Client
  2. JCBC-270

client does not handle failure of EPT node with memcached bucket

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: 1.1.2
    • Fix Version/s: 1.1.7
    • Component/s: Core
    • Security Level: Public
    • Labels:
      None
    • Environment:
      3 node cluster of 2.0, one memcached bucket authenticated, Couchbase Java Client 1.1.2 with required spymemcached 2.8.11

      Description

      When investigating a case, it seems that the dropped configuration when using a memcached bucket is never reestablished. I'm not sure if this is because we rely on timeouts (which we won't get in this case) or something else.

      Steps to reproduce:
      1. Set up three node cluster
      2. Run a constant workload, starting off of one of the nodes (192.168.1.200 in my config)
      3. Remove that node from the cluster

      Observed behavior:
      The client sees the config dropped, but doesn't reconfigure.

      Expected behavior:
      Client bootstraps off of one of the other nodes.

      Attached file shows the config log in this case.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        daschl Michael Nitschinger added a comment -

        Hey Matt,

        one quick update.. I don't know yet if its related, but you've been using the wrong netty version.. your logs show /Users/ingenthr/lib/netty-3.2.5.Final.jar' but the correct one is 3.5.5!

        Since netty handles the streaming connection, this may be related - I dont know yet.

        Show
        daschl Michael Nitschinger added a comment - Hey Matt, one quick update.. I don't know yet if its related, but you've been using the wrong netty version.. your logs show /Users/ingenthr/lib/netty-3.2.5.Final.jar' but the correct one is 3.5.5! Since netty handles the streaming connection, this may be related - I dont know yet.
        Hide
        daschl Michael Nitschinger added a comment -

        Matt, with the change proposed in JCBC-271 and using Netty 3.5.5, I dont see this behaviour (anymore). I tried with 3.2.5 but my connections always go from unbound to connected after some time, even when I wildly failover/rebalance - as it should be. It even waits as we implemented it when I remove both EPT nodes for some time and then add them back it comes back nicely.

        Can you try to repro with those two changes and if it still fails lets do a quick screen sharing.

        Show
        daschl Michael Nitschinger added a comment - Matt, with the change proposed in JCBC-271 and using Netty 3.5.5, I dont see this behaviour (anymore). I tried with 3.2.5 but my connections always go from unbound to connected after some time, even when I wildly failover/rebalance - as it should be. It even waits as we implemented it when I remove both EPT nodes for some time and then add them back it comes back nicely. Can you try to repro with those two changes and if it still fails lets do a quick screen sharing.
        Hide
        ingenthr Matt Ingenthron added a comment -

        Indeed, I could not reproduce this one after moving to the right dependencies.

        Show
        ingenthr Matt Ingenthron added a comment - Indeed, I could not reproduce this one after moving to the right dependencies.

          People

          • Assignee:
            daschl Michael Nitschinger
            Reporter:
            ingenthr Matt Ingenthron
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes