Critical Description
32 backup binary merge tests are failing in the latest RC3 build (7.2.1-5928). They did not fail in 7.2.1-5904 ("only" 3 tests failed there).
An example for a failure:
['Traceback (most recent call last):\n File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 1863, in validate_backup_data\n .format(bucket.name))\nException: Data not match in backup bucket standard_bucket0\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "pytests/ent_backup_restore/enterprise_backup_merge_tests.py", line 39, in test_multiple_backups_merges\n self.do_backup_merge_actions()\n File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 4220, in do_backup_merge_actions\n self.backup_merge_actions[action](self)\n File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 3406, in merge\n self.backup_merge_validate(repeats=self.number_of_repeats, skip_validation=skip_validation)\n File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 1717, in backup_merge_validate\n self.validate_backup_data(**kwargs)\n File "pytests/ent_backup_restore/enterprise_backup_restore_base.py", line 1871, in validate_backup_data\n raise Exception("\\nMissing key: {0}".format(key))\nException: \nMissing key: ent-backup26\ |
A link to the Jenkins job:
http://qa.sc.couchbase.com/job/test_suite_executor/609194/testReport/
I have attached the console output of a local run I did as well as cbbackupmgr logs (not necessarily of the same run):
./testrunner -i ~/work/ini/neo.ini -p get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,skip_log_scan=True,force-updates=True,get-cbcollect-info=False,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 -t ent_backup_restore.enterprise_backup_merge_tests.EnterpriseBackupMergeTest.test_multiple_backups_merges,actions="backup_with_ops:1,backup:1,rebalance,backup_with_ops:1,backup:1,merge:1&4",nodes_in=1,document_type=binary,GROUP=P0,get-coredumps=False,force_kill_memached=True,verify_unacked_bytes=True,skip_log_scan=True,force-updates=True,get-cbcollect-info=False,bucket_storage=couchstore,sirius_url=http://172.23.120.103:4000 |
It is possible that it's another instance of MB-58216, which is a known test issues (as mentioned in the comments)