Uploaded image for project: 'Spymemcached Java Client'
  1. Spymemcached Java Client
  2. SPY-89

invalid binary protocol server response during rebalance


    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.1
    • Fix Version/s: .next
    • Component/s: library
    • Security Level: Public
    • Labels:


      When investigating an issue, I found that the server would unexpectedly respond with a 0x0a command for optimized get requests. This seems to be in violation of the protocol, and it causes all kinds of havoc for a client library.

      The scenario is spymemcached's internals doing an optimized get. It's reading the response for an operation during a rebalance. It has been observed with both the node being rebalanced into the cluster and with a node leaving the cluster.

      Reading the minimum header, the client library is checking the magic, and then the response command. It's expecting a 0x00, but receiving a 0x0a.

      I poked through the server side code, and 0x0a corresponds to TAP_NOOP. I poked around a little further, and I don't see any situation where we would respond with this 0x0a to a non-TAP client.

      The assertion this is raising is right here:

      If I force spymemcached to not optimize, the problem goes away.

      I will attach a packet capture.

      1. rebalanceout.out
        6.10 MB
        Matt Ingenthron
      2. rebalanceout-notes.txt
        0.3 kB
        Matt Ingenthron
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.


        ingenthr Matt Ingenthron created issue -
        ingenthr Matt Ingenthron made changes -
        Field Original Value New Value
        Attachment rebalanceout.out [ 13411 ]
        Attachment rebalanceout-notes.txt [ 13412 ]
        farshid Farshid Ghods (Inactive) made changes -
        Fix Version/s 2.0-developer-preview-5 [ 10290 ]
        trond Trond Norbye made changes -
        Assignee Damien Katz [ damien ] Matt Ingenthron [ ingenthr ]
        Component/s clients [ 10042 ]
        Component/s storage-engine [ 10175 ]
        ingenthr Matt Ingenthron made changes -
        Project Couchbase Server [ 10010 ] Spymemcached Java Client [ 10047 ]
        Key MB-5410 SPY-89
        Affects Version/s 2.8.1 [ 10302 ]
        Affects Version/s 1.8.0 [ 10248 ]
        Fix Version/s 2.0-developer-preview-5 [ 10290 ]
        Component/s library [ 10120 ]
        Component/s clients [ 10042 ]
        daschl Michael Nitschinger made changes -
        Fix Version/s .next [ 10205 ]


          • Assignee:
            ingenthr Matt Ingenthron
            ingenthr Matt Ingenthron
          • Votes:
            0 Vote for this issue
            2 Start watching this issue


            • Created:

              Gerrit Reviews

              There are no open Gerrit changes