{UPR}:: MemcachedError: Memcached error #7 'Not my vbucket': Error during Rebalance-in

3.0.0-846

1. Create 7 Node cluster
2. Create Default Bucket
3. Add 500 Items
4. Rebalance-out 1 Node with mutations in parallel
5. Rebalance-in 1 Node with mutations in parallel
6. Rebalance-out 2 Node with mutations in parallel
7. Rebalance-in 2 Node with mutations in parallel
8. Rebalance-out 3 Node with mutations in parallel
9. Rebalance-in 3 Node with mutations in parallel

In Step 9 we hit the following exception

MemcachedError: Memcached error #7 'Not my vbucket': {"rev":5804,"name":"default","uri":"/pools/default/buckets/default?bucket_uuid=41c59b28b2ee111b363566a38edc674c","streamingUri":"/pools/default/bucketsStreaming/default?bucket_uuid=41c59b28b2ee111b363566a38edc674c","nodes":[{"couchApiBase":"http://172.23.107.146:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.146:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://172.23.107.147:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.147:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://172.23.107.148:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.148:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://172.23.107.149:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.149:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://172.23.107.150:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.150:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://172.23.107.151:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.151:8091","ports":{"proxy":11211,"direct":11210}},{"couchApiBase":"http://172.23.107.152:8092/default%2B41c59b28b2ee111b363566a38edc674c","hostname":"172.23.107.152:8091","ports":{"proxy":11211,"direct":11210}}],"nodesExt":[{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.146"},{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.147"},{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.148"},{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.149"},{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.150"},{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.151"},{"services":

{"mgmt":8091,"capi":8092,"moxi":11211,"kv":11210,"kvSSL":11207,"capiSSL":18092,"mgmtSSL":18091}

,"hostname":"172.23.107.152"}],"nodeLocator":"vbucket","uuid":"41c59b28b2ee111b363566a38edc674c","ddocs":

{"uri":"/pools/default/buckets/default/ddocs"}

,"vBucketServerMap":

{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["172.23.107.146:11210","172.23.107.147:11210","172.23.107.148:11210","172.23.107.149:11210","172.23.107.150:11210","172.23.107.151:11210","172.23.107.152:11210"],"vBucketMap":[[0,1],[0,1],[0,1],[0,1],[0,2],[0,2],[0,2],[0,3],[0,3],[0,3],[0,4],[0,4],[0,4],[0,5],[0,5],[0,5],[0,3],[0,3],[0,3],[1,0],[1,0],[1,0],[1,0],[1,2],[1,2],[1,2],[1,3],[1,3],[1,3],[1,4],[1,4],[1,4],[1,5],[1,2],[1,2],[1,2],[1,2],[1,2],[2,0],[2,0],[2,0],[2,1],[2,1],[2,1],[2,3],[2,3],[2,3],[2,4],[2,4],[2,4],[2,5],[2,5],[2,5],[2,6],[2,6],[2,1],[3,0],[3,0],[3,0],[3,1],[3,1],[3,1],[3,2],[3,2],[3,2],[3,4],[3,4],[3,4],[3,5],[3,5],[3,5],[3,6],[2,3],[2,3],[0,1],[0,1],[0,1],[4,1],[4,1],[1,0],[4,2],[4,2],[4,2],[4,3],[4,3],[3,0],[4,5],[4,5],[4,5],[2,3],[3,1],[3,1],[5,0],[5,0],[0,1],[1,0],[1,0],[1,0],[5,2],[5,2],[5,2],[5,3],[5,3],[5,3],[5,4],[5,4],[5,4],[3,1],[3,2],[3,2],[0,1],[0,2],[0,2],[1,0],[1,2],[1,2],[2,0],[2,0],[2,1],[6,3],[6,3],[6,3],[6,4],[6,4],[6,4],[6,5],[6,5],[3,2]],"vBucketMapForward":[[0,1],[0,1],[0,1],[0,1],[0,2],[0,2],[0,2],[0,3],[0,3],[0,3],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,3],[0,3],[0,3],[1,0],[1,0],[1,0],[1,0],[1,2],[1,2],[1,2],[1,3],[1,3],[1,3],[0,3],[0,3],[0,3],[0,3],[1,2],[1,2],[1,2],[1,2],[1,2],[2,0],[2,0],[2,0],[2,1],[2,1],[2,1],[2,3],[2,3],[2,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[2,1],[3,0],[3,0],[3,0],[3,1],[3,1],[3,1],[3,2],[3,2],[3,2],[2,1],[2,1],[2,1],[2,1],[2,1],[2,3],[2,3],[2,3],[2,3],[0,1],[0,1],[0,1],[1,0],[1,0],[1,0],[2,0],[2,0],[2,0],[3,0],[3,0],[3,0],[2,3],[2,3],[2,3],[2,3],[3,1],[3,1],[0,1],[0,1],[0,1],[1,0],[1,0],[1,0],[2,0],[2,0],[2,0],[3,0],[3,0],[3,0],[3,1],[3,1],[3,1],[3,1],[3,2],[3,2],[0,1],[0,2],[0,2],[1,0],[1,2],[1,2],[2,0],[2,0],[2,1],[3,0],[3,1],[3,1],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2]]}

,"bucketCapabilitiesVer":"","bucketCapabilities":["cbhello","touch","couchapi","cccp"]} for vbucket :92 to mc 172.23.107.146:11210

Test Case: /testrunner -i palm.ini get-cbcollect-info=False,get-logs=False,stop-on-failure=False,get-coredumps=True,force_kill_memached=False,verify_unacked_bytes=True -t rebalance.rebalanceinout.RebalanceInOutTests.incremental_rebalance_in_out_with_mutation_and_deletion,items=500000,value_size=256,max_verify=100000,GROUP=IN_OUT;P0

https://s3.amazonaws.com/bugdb/jira/MB-11497/rebalance_out_vbucket_not_found_3_0_0_846.tar.gz