GoXDCR: mass_vbopaque_check fails constantly due to UUID format mismatch

Build

---

4.0.0-1977

Found while analyzing system test logs on node .47 . MassValidateVBUUID failed constantly.

CheckpointManager 2015-04-29T11:37:53.720-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:37:55.124-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:37:55.478-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:38:53.719-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:38:55.120-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:38:55.482-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:39:53.718-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:39:55.122-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:39:55.486-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
:
:
:
CheckpointManager 2015-04-29T11:57:53.751-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:57:55.925-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383
CheckpointManager 2015-04-29T11:58:11.594-07:00 [ERROR] MassValidateVBUUID failed, err=_commit_for_checkpoint retruns an invalid response. element in mismatch field should have format of [vbno, vbuuid], it is 383

Attaching logs from .47.

From couchdb.log on .57(from target nozzle IP), I see commit_for_checkpoint returning response in format [vbuuid,vbno]. Please note mismatch in format.

ns_couchdb.log:[xdcr:debug,2015-04-29T12:06:54.777,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{81042509407263,208}

, VBOpaque: 81042509407263
ns_couchdb.log:[couchdb:info,2015-04-29T12:06:54.777,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:07:08.807,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{186829605166375,199}

, VBOpaque: 186829605166375
ns_couchdb.log:[couchdb:info,2015-04-29T12:07:08.807,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:07:22.870,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{51055050249273,164}

, VBOpaque: 51055050249273
ns_couchdb.log:[couchdb:info,2015-04-29T12:07:22.870,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:07:36.932,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{268111949941893,150}

, VBOpaque: 268111949941893
ns_couchdb.log:[couchdb:info,2015-04-29T12:07:36.932,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:07:50.994,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{264504734406647,173}

, VBOpaque: 264504734406647
ns_couchdb.log:[couchdb:info,2015-04-29T12:07:50.994,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:08:05.058,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{216036508134626,151}

, VBOpaque: 216036508134626
ns_couchdb.log:[couchdb:info,2015-04-29T12:08:05.058,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:08:19.118,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{242859019577258,194}

, VBOpaque: 242859019577258
ns_couchdb.log:[couchdb:info,2015-04-29T12:08:19.118,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:08:33.185,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{6702970611195,199}

, VBOpaque: 6702970611195
ns_couchdb.log:[couchdb:info,2015-04-29T12:08:33.185,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200
ns_couchdb.log:[xdcr:debug,2015-04-29T12:08:47.246,couchdb_ns_1@127.0.0.1:<0.19096.0>:capi_replication:handle_commit_for_checkpoint:408]UUID:

{32469056073261,212}

, VBOpaque: 32469056073261
ns_couchdb.log:[couchdb:info,2015-04-29T12:08:47.246,couchdb_ns_1@127.0.0.1:<0.19096.0>:couch_log:info:41]172.23.105.47 - - POST /_commit_for_checkpoint 200