Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47095

6.0.3 couchbase backup fails to restore in 7.1.0

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.1.0
    • 7.1.0
    • tools
    • 7.1.0-1027
    • Untriaged
    • 1
    • Unknown

    Description

      original version of couchbase in which the backup was taken : 6.0.3
      version of couchbase used in the test ; 7.1.0-1027 
      error seen in the test : 

      08:07:16 2021-06-21T19:37:17 [INFO] Running: /opt/couchbase/bin/cbbackupmgr restore --archive /fts/backup --repo fts_geo --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes
      08:07:18
      08:07:18 Fatal error: run() received nonzero return code 1 while executing!
      08:07:18
      08:07:18 Requested: /opt/couchbase/bin/cbbackupmgr restore --archive /fts/backup --repo fts_geo --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes
      08:07:18 Executed: /bin/bash -l -c -o pipefail "/opt/couchbase/bin/cbbackupmgr restore --archive /fts/backup --repo fts_geo --threads 24 --cluster http://localhost:8091 --username Administrator --password password --disable-ft-indexes --disable-gsi-indexes"
      08:07:18
      08:07:18 =============================== Standard output ===============================
      08:07:18
      08:07:18 Error restoring cluster: cannot read backup config: backup config at '/fts/backup/exportFiles/backup-meta.json' not found
      08:07:18
      08:07:18 ================================================================================
      08:07:18
      08:07:18 Aborting. 

      Link to the job: http://perf.jenkins.couchbase.com/view/FTS/job/atlas-sdk/13622/
      Link to the cbcollect logs:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-13622/172.23.99.211.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-13622/172.23.99.39.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-atlas-sdk-13622/172.23.99.40.zip

      Attachments

        For Gerrit Dashboard: MB-47095
        # Subject Branch Project Status CR V

        Activity

          jyotsna.nayak Jyotsna Nayak created issue -
          pvarley Patrick Varley made changes -
          Field Original Value New Value
          Component/s tools [ 10223 ]
          Component/s fts [ 12726 ]

          Jyotsna Nayak Would it be possible to get the cbbackupmgr collect-logs please.

          pvarley Patrick Varley added a comment - Jyotsna Nayak Would it be possible to get the cbbackupmgr collect-logs please.

          I logged on to the machine, it looks like some one has create a folder in the backup repo called exportFiles. cbbackupmgr expects everything under the archive to be under it's control. By creating this folder cbbackupmgr believes it's a new repo, however when it tries to read the repo metadata it fails as expected with the error: cannot read backup config: backup config at '/fts/backup/exportFiles/backup-meta.json' not found

          As we can see exportFiles is not a backup repo

          file list of

          # ls -alR /fts/backup/exportFiles/
          /fts/backup/exportFiles/:
          total 46041104
          drwxr-xr-x.  2 root root        4096 Feb 18 19:50 .
          drwxr-xr-x. 10 root root        4096 Jun 22 20:46 ..
          -rw-rw----.  1 root root 42911442534 Jan 22 09:29 fts_generic_20M.json
          -rw-r-----.  1 root root        4096 Feb 18 19:50 .fts_generic_20M.json.swp
          -rw-r--r--.  1 root root  2149663127 Oct 20  2020 fts_wiki.json
          -rw-r--r--.  1 root root        2499 Jan 22 09:50 splitData.py
          -rw-r--r--.  1 root root  2084941947 Feb  2 21:38 temp.json
          

          I would recommend not messing with the archive filesystem layout and moving exportFiles out of the archive.

          I will check in case cbbackupmgr has changed behaviour here, it seems strange to fail on a repo that is not being actioned on.

          That being said I don't think this is a bug.

          pvarley Patrick Varley added a comment - I logged on to the machine, it looks like some one has create a folder in the backup repo called exportFiles . cbbackupmgr expects everything under the archive to be under it's control. By creating this folder cbbackupmgr believes it's a new repo, however when it tries to read the repo metadata it fails as expected with the error: cannot read backup config: backup config at '/fts/backup/exportFiles/backup-meta.json' not found As we can see exportFiles is not a backup repo file list of # ls -alR /fts/backup/exportFiles/ /fts/backup/exportFiles/: total 46041104 drwxr-xr-x. 2 root root 4096 Feb 18 19:50 . drwxr-xr-x. 10 root root 4096 Jun 22 20:46 .. -rw-rw----. 1 root root 42911442534 Jan 22 09:29 fts_generic_20M.json -rw-r-----. 1 root root 4096 Feb 18 19:50 .fts_generic_20M.json.swp -rw-r--r--. 1 root root 2149663127 Oct 20 2020 fts_wiki.json -rw-r--r--. 1 root root 2499 Jan 22 09:50 splitData.py -rw-r--r--. 1 root root 2084941947 Feb 2 21:38 temp.json I would recommend not messing with the archive filesystem layout and moving exportFiles out of the archive. I will check in case cbbackupmgr has changed behaviour here, it seems strange to fail on a repo that is not being actioned on. That being said I don't think this is a bug.
          james.lee James Lee made changes -
          Assignee Patrick Varley [ pvarley ] James Lee [ james.lee ]
          james.lee James Lee made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          james.lee James Lee added a comment -

          Hi Jyotsna Nayak,

          Patrick Varley and I discussed this issue and we agreed that we dislike that an invalid/corrupted repository can affect a sub-command running using a completely different repository. This behavior has changed in 7.1.0; it was introduced in this change (more specifically, this line).

          With that being said, the archives created by 'cbbackupmgr' should not be modified in any way as discussed in the documentation so I think we can also agree that this is technically undefined behavior so if possible, you shouldn't rely on this fix completely resolving any issues when using this archive (as described below).

          I've got a patch up against 'backup' which should resolve this issue by only mounting repositories which are going to be used (where applicable). However, please note that other sub-commands (which require all repositories to be mounted) will fail; this is to be expected.

          Thanks,
          James

          james.lee James Lee added a comment - Hi Jyotsna Nayak , Patrick Varley and I discussed this issue and we agreed that we dislike that an invalid/corrupted repository can affect a sub-command running using a completely different repository. This behavior has changed in 7.1.0; it was introduced in this change (more specifically, this line). With that being said, the archives created by ' cbbackupmgr ' should not be modified in any way as discussed in the documentation so I think we can also agree that this is technically undefined behavior so if possible, you shouldn't rely on this fix completely resolving any issues when using this archive (as described below). I've got a patch up against ' backup ' which should resolve this issue by only mounting repositories which are going to be used (where applicable). However, please note that other sub-commands (which require all repositories to be mounted) will fail; this is to be expected. Thanks, James
          raju Raju Suravarjjala made changes -
          Affects Version/s Neo [ 17615 ]
          raju Raju Suravarjjala made changes -
          Fix Version/s Neo [ 17615 ]
          jyotsna.nayak Jyotsna Nayak added a comment - - edited

          James Lee , Patrick Varley , thank you ! I have removed the extra folder from the archive folder. 

          The bucket was restored with the documents after this.
          Shall I close this issue ?

           

          jyotsna.nayak Jyotsna Nayak added a comment - - edited James Lee  , Patrick Varley  , thank you ! I have removed the extra folder from the archive folder.  The bucket was restored with the documents after this. Shall I close this issue ?  
          owend Daniel Owen added a comment -

          Hi Jyotsna Nayak,

          James is out of office, however I think we should keep the ticket open until he is back - as he has a patch to submit.

          thanks

          owend Daniel Owen added a comment - Hi Jyotsna Nayak , James is out of office, however I think we should keep the ticket open until he is back - as he has a patch to submit. thanks
          james.lee James Lee made changes -
          Assignee James Lee [ james.lee ] Jyotsna Nayak [ jyotsna.nayak ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Resolved [ 5 ]

          Build couchbase-server-7.1.0-1064 contains backup commit 5922707 with commit message:
          MB-47095 Where applicable only mount required repositories

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1064 contains backup commit 5922707 with commit message: MB-47095 Where applicable only mount required repositories
          mihir.kamdar Mihir Kamdar (Inactive) made changes -
          Assignee Jyotsna Nayak [ jyotsna.nayak ] Thuan Nguyen [ thuan ]
          thuan Thuan Nguyen added a comment -

          Verified on build 7.1.0-2469. Backup repo in 6.0.3 was used to restore with cbbackupmgr in build 7.1.0-2469

          [root@s44015 ~]# /opt/couchbase/bin/cbbackupmgr restore --auto-create-buckets -c localhost -u Administrator -p password -a /tmp/bk/ -r bk
          Restoring backup '2022-03-11T09_41_00.896860342-08_00'
          Copied all data in 6.936s (Avg. 29.04KiB/Sec)                                                                                                                                                                                             586 items / 174.26KiB
          [==================================================================================================================================================================================================================================================== ] 100.00%
           
          | Transfer
          | --------
          | Status    | Avg Transfer Rate | Started At                      | Finished At                     | Duration |
          | Succeeded | 29.04KiB/s        | Fri, 11 Mar 2022 09:47:27 -0800 | Fri, 11 Mar 2022 09:47:34 -0800 | 6.946s   |
           
          | Bucket
          | ------
          | Name           | Status    | Transferred | Avg Transfer Rate | Started At                      | Finished At                     | Duration |
          | gamesim-sample | Succeeded | 174.26KiB   | 174.26KiB/s       | Fri, 11 Mar 2022 09:47:33 -0800 | Fri, 11 Mar 2022 09:47:34 -0800 | 1.174s   |
          |
          | Mutations                    | Deletions                    | Expirations                  |
          | ---------                    | ---------                    | -----------                  |
          | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped |
          | 586      | 0       | 0       | 0        | 0       | 0       | 0        | 0       | 0       |
           
          Restore completed successfully
          [root@s44015 ~]# 
          
          

          thuan Thuan Nguyen added a comment - Verified on build 7.1.0-2469. Backup repo in 6.0.3 was used to restore with cbbackupmgr in build 7.1.0-2469 [root@s44015 ~]# /opt/couchbase/bin/cbbackupmgr restore --auto-create-buckets -c localhost -u Administrator -p password -a /tmp/bk/ -r bk Restoring backup '2022-03-11T09_41_00.896860342-08_00' Copied all data in 6.936s (Avg. 29.04KiB/Sec) 586 items / 174.26KiB [==================================================================================================================================================================================================================================================== ] 100.00%   | Transfer | -------- | Status | Avg Transfer Rate | Started At | Finished At | Duration | | Succeeded | 29.04KiB/s | Fri, 11 Mar 2022 09:47:27 -0800 | Fri, 11 Mar 2022 09:47:34 -0800 | 6.946s |   | Bucket | ------ | Name | Status | Transferred | Avg Transfer Rate | Started At | Finished At | Duration | | gamesim-sample | Succeeded | 174.26KiB | 174.26KiB/s | Fri, 11 Mar 2022 09:47:33 -0800 | Fri, 11 Mar 2022 09:47:34 -0800 | 1.174s | | | Mutations | Deletions | Expirations | | --------- | --------- | ----------- | | Received | Errored | Skipped | Received | Errored | Skipped | Received | Errored | Skipped | | 586 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |   Restore completed successfully [root@s44015 ~]#
          thuan Thuan Nguyen made changes -
          Sprint Status Current Sprint [ 10027 ]
          Status Resolved [ 5 ] Closed [ 6 ]

          People

            thuan Thuan Nguyen
            jyotsna.nayak Jyotsna Nayak
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty