Uploaded image for project: 'Spymemcached Java Client'
  1. Spymemcached Java Client
  2. SPY-89

invalid binary protocol server response during rebalance


    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.8.1
    • Fix Version/s: .next
    • Component/s: library
    • Security Level: Public
    • Labels:


      When investigating an issue, I found that the server would unexpectedly respond with a 0x0a command for optimized get requests. This seems to be in violation of the protocol, and it causes all kinds of havoc for a client library.

      The scenario is spymemcached's internals doing an optimized get. It's reading the response for an operation during a rebalance. It has been observed with both the node being rebalanced into the cluster and with a node leaving the cluster.

      Reading the minimum header, the client library is checking the magic, and then the response command. It's expecting a 0x00, but receiving a 0x0a.

      I poked through the server side code, and 0x0a corresponds to TAP_NOOP. I poked around a little further, and I don't see any situation where we would respond with this 0x0a to a non-TAP client.

      The assertion this is raising is right here:

      If I force spymemcached to not optimize, the problem goes away.

      I will attach a packet capture.

      1. rebalanceout.out
        6.10 MB
        Matt Ingenthron
      2. rebalanceout-notes.txt
        0.3 kB
        Matt Ingenthron
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.


        There are no open Gerrit changes for this issue.


          • Assignee:
            ingenthr Matt Ingenthron
            ingenthr Matt Ingenthron
          • Votes:
            0 Vote for this issue
            2 Start watching this issue


            • Created:

              Gerrit Reviews

              There are no open Gerrit changes