Observe (and possibly other) callbacks receiving NULL callback more than once when batched

See here http://paste.scsys.co.uk/219902 for an example.

This primarily affects observe operation in 'multi' mode; where multiple observe requests are queued up. This fails when the request distribution is asymmetric.

This also affects stats and version commands if coupled with other commands in an asynchronous mode.

This will definitely affect node.js.

In summary, the function lcb_lookup_server_with_packet tries to find 'associated' packets; but it does it wrongly. It assumes associated packets will always be at the tail (or pop-end) of the command log.

However this will only be the case in a purely synchronous mode where there are no subsequent commands. The following scenario demonstrates this rather well:

With three servers and two keys (and one replica), you have something like this for the vb/server map

"K1" = servers[1], servers[2]
"K2" = servers[0], servers[1]

When the commands are sent out, the command look for the server looks something like:

– order in the buffer
BUFFERS:
[ OLD .. NEW]

0: "K2",
1: "K1", "K2"
2: "K1",

LOGICAL PRECEDENCE:
[ OLD .. NEW ]

"K1", "K2"

The idea to keep in mind is that even though we've scheduled the request for "K1" first, we may very well receive the response for "K2" first - this is especially true if server[0] is quicker than the other servers.

So what happens when "K2" is received first? - the code checks to see the older commands on all the other servers (only the oldest command), and if not, it assumes this is the last command to be received and notifies the user that the command has finished.

So now the buffers look like:

0: "K3"
1: "K1", "K2"
2: "K1"

Now, server[1] responds with "K1". Since the request for "K1" is still at the pop-end of server[2] nothing special happens here

The buffers look like:

0:
1: "K2"
2: "K1"

Then server[1] responds again, this time with the next response for "K2". It delivers the callback again. In a typical use case a user maintains a cookie tied to the command, and when the command is done, the cookie is freed. Plainly speaking, this double-singalling (i.e. telling the user a command has finished twice) segfaults the app.