Details
Major Description
original version of couchbase in which the backup was taken : 6.0.3
version of couchbase used in the test ; 7.1.0-1027
error seen in the test :
08:07:16 2021-06-21T19:37:17 [INFO] Running: /opt/couchbase/bin/cbbackupmgr restore --archive /fts/backup --repo fts_geo --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes
08:07:18
08:07:18 Fatal error: run() received nonzero return code 1 while executing!
08:07:18
08:07:18 Requested: /opt/couchbase/bin/cbbackupmgr restore --archive /fts/backup --repo fts_geo --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes
08:07:18 Executed: /bin/bash -l -c -o pipefail "/opt/couchbase/bin/cbbackupmgr restore --archive /fts/backup --repo fts_geo --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes"
08:07:18
08:07:18 =============================== Standard output ===============================
08:07:18
08:07:18 Error restoring cluster: cannot read backup config: backup config at '/fts/backup/exportFiles/backup-meta.json' not found
08:07:18
08:07:18 ================================================================================
08:07:18
08:07:18 Aborting.
Link to the job: http://perf.jenkins.couchbase.com/view/FTS/job/atlas-sdk/13622/
Link to the cbcollect logs:
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-13622/172.23.99.211.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-13622/172.23.99.39.zip
https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-13622/172.23.99.40.zip
Attachments
Activity
Build couchbase-server-7.1.0-1064 contains backup commit 5922707 with commit message:
MB-47095 Where applicable only mount required repositories
Hi Jyotsna Nayak,
James is out of office, however I think we should keep the ticket open until he is back - as he has a patch to submit.
thanks
James Lee , Patrick Varley , thank you ! I have removed the extra folder from the archive folder.
The bucket was restored with the documents after this.
Shall I close this issue ?
Hi Jyotsna Nayak,
Patrick Varley and I discussed this issue and we agreed that we dislike that an invalid/corrupted repository can affect a sub-command running using a completely different repository. This behavior has changed in 7.1.0; it was introduced in this change (more specifically, this line).
With that being said, the archives created by 'cbbackupmgr' should not be modified in any way as discussed in the documentation so I think we can also agree that this is technically undefined behavior so if possible, you shouldn't rely on this fix completely resolving any issues when using this archive (as described below).
I've got a patch up against 'backup' which should resolve this issue by only mounting repositories which are going to be used (where applicable). However, please note that other sub-commands (which require all repositories to be mounted) will fail; this is to be expected.
Thanks,
James
I logged on to the machine, it looks like some one has create a folder in the backup repo called exportFiles. cbbackupmgr expects everything under the archive to be under it's control. By creating this folder cbbackupmgr believes it's a new repo, however when it tries to read the repo metadata it fails as expected with the error: cannot read backup config: backup config at '/fts/backup/exportFiles/backup-meta.json' not found
As we can see exportFiles is not a backup repo
|
file list of |
# ls -alR /fts/backup/exportFiles/
|
/fts/backup/exportFiles/:
|
total 46041104
|
drwxr-xr-x. 2 root root 4096 Feb 18 19:50 .
|
drwxr-xr-x. 10 root root 4096 Jun 22 20:46 ..
|
-rw-rw----. 1 root root 42911442534 Jan 22 09:29 fts_generic_20M.json
|
-rw-r-----. 1 root root 4096 Feb 18 19:50 .fts_generic_20M.json.swp
|
-rw-r--r--. 1 root root 2149663127 Oct 20 2020 fts_wiki.json
|
-rw-r--r--. 1 root root 2499 Jan 22 09:50 splitData.py
|
-rw-r--r--. 1 root root 2084941947 Feb 2 21:38 temp.json
|
I would recommend not messing with the archive filesystem layout and moving exportFiles out of the archive.
I will check in case cbbackupmgr has changed behaviour here, it seems strange to fail on a repo that is not being actioned on.
That being said I don't think this is a bug.
Jyotsna Nayak Would it be possible to get the cbbackupmgr collect-logs please.
Verified on build 7.1.0-2469. Backup repo in 6.0.3 was used to restore with cbbackupmgr in build 7.1.0-2469
[root@s44015 ~]# /opt/couchbase/bin/cbbackupmgr restore --auto-create-buckets -c localhost -u Administrator -p password -a /tmp/bk/ -r bkRestoring backup '2022-03-11T09_41_00.896860342-08_00'Copied all data in 6.936s (Avg. 29.04KiB/Sec) 586 items / 174.26KiB[==================================================================================================================================================================================================================================================== ] 100.00%| Transfer| --------| Status | Avg Transfer Rate | Started At | Finished At | Duration || Succeeded | 29.04KiB/s | Fri, 11 Mar 2022 09:47:27 -0800 | Fri, 11 Mar 2022 09:47:34 -0800 | 6.946s || Bucket| ------| Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration || gamesim-sample | Succeeded | 174.26KiB | 174.26KiB/s | Fri, 11 Mar 2022 09:47:33 -0800 | Fri, 11 Mar 2022 09:47:34 -0800 | 1.174s ||| Mutations | Deletions | Expirations || --------- | --------- | ----------- || Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped || 586 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |Restore completed successfully[root@s44015 ~]#