Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48744

cbbackupmgr backup failed on 7.1.0-1401

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Neo
    • Neo
    • tools
    • Untriaged
    • 1
    • Yes

    Description

      Our backup performance runs failed on 7.1.0-1401. The issue is reproducible. The latest good run was running on build 7.1.0-1345.

      http://perf.jenkins.couchbase.com/job/rhea-5node2/1490/

      Running: ./opt/couchbase/bin/cbbackupmgr backup --archive /workspace/backup --repo default --host http://172.23.97.26 --username Administrator --password password --threads 16 --storage rift

      Fatal error: local() encountered an error (return code 1) while executing './opt/couchbase/bin/cbbackupmgr backup --archive /workspace/backup --repo default --host http://172.23.97.26 --username Administrator --password password --threads 16 --storage rift'

       

      From backup-0.log: 

      2021-10-03T11:20:59.371-07:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint '/pools/default/buckets/bucket-1/ddocs' failed with status code 400 – logging.(*ToolsCommonLogger).Log() at tools_common.go:28

      2021-10-03T11:20:59.371-07:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket-1': failed to transfer index definitions for bucket 'bucket-1': failed to transfer views: failed to get view definitions: failed to execute request: unexpected status code 400 for 'GET' request to '/pools/default/buckets/bucket-1/ddocs', {"error":"no_ddocs_service"}

       
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T164633-ns_1%40172.23.97.26.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T164633-ns_1%40172.23.97.27.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T164633-ns_1%40172.23.97.28.zip
      https://s3.amazonaws.com/bugdb/jira/qe/collectinfo-2021-10-04T164633-ns_1%40172.23.97.29.zip
       

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-48744
          # Subject Branch Project Status CR V

          Activity

            owend Daniel Owen added a comment -

            Hey James Lee could you take a look?

            unexpected status code 400 for 'GET' request to '/pools/default/buckets/bucket-1/ddocs', {"error":"no_ddocs_service"}
            

            seems like the server REST endpoint is having issues.

            owend Daniel Owen added a comment - Hey James Lee could you take a look? unexpected status code 400 for 'GET' request to '/pools/default/buckets/bucket-1/ddocs', {"error":"no_ddocs_service"} seems like the server REST endpoint is having issues.
            james.lee James Lee added a comment -

            I suspect this is caused by MB-47740, we'll need to add some specific handling for this.

            Looking at the logs, I see:

            Buckets

              Bucket            Type Quota (MB) Prio Evict Indx Reps Conf Cmpct Purge (Days) Access Flush Max TTL Compress Storage   Prod   Dev
              ---------------------------------------------------------------------------------------------------------------------------------
              bucket-1            CB      10240  Low  Full       Off  Seq  100%  3 (default)   SASL    On     Off  Passive   Magma    0/0   0/0
              Total (1 buckets)    -      10240    -     -         -    -     -            -      -     -       -        -       -    0/0   0/0
            

            As we can see, this bucket is using the magma backend; which as of 7.1.0-1397 will return an error when users attempt to use/access views.

            james.lee James Lee added a comment - I suspect this is caused by MB-47740 , we'll need to add some specific handling for this. Looking at the logs, I see: Buckets Bucket Type Quota (MB) Prio Evict Indx Reps Conf Cmpct Purge (Days) Access Flush Max TTL Compress Storage Prod Dev --------------------------------------------------------------------------------------------------------------------------------- bucket-1 CB 10240 Low Full Off Seq 100% 3 (default) SASL On Off Passive Magma 0/0 0/0 Total (1 buckets) - 10240 - - - - - - - - - - - 0/0 0/0 As we can see, this bucket is using the magma backend; which as of 7.1.0-1397 will return an error when users attempt to use/access views.
            owend Daniel Owen added a comment -

            ah good spot yes we need to cover this case.

            owend Daniel Owen added a comment - ah good spot yes we need to cover this case.

            Do we know why magma does not support view? There are some DCP features that only views and backup use and I want to ensure that is not the case here.

            pvarley Patrick Varley added a comment - Do we know why magma does not support view? There are some DCP features that only views and backup use and I want to ensure that is not the case here.
            james.lee James Lee added a comment -

            I'm not sure, MB-47740 doesn't provide a reason/document explaining the decision; I imagine that'd be a question best answered by PM/the Magma team.

            Regarding the DCP features, I think we should be fine; I spoke to Ben from KV and DCP Disk only mode should work as expected.

            james.lee James Lee added a comment - I'm not sure, MB-47740 doesn't provide a reason/document explaining the decision; I imagine that'd be a question best answered by PM/the Magma team. Regarding the DCP features, I think we should be fine; I spoke to Ben from KV and DCP Disk only mode should work as expected.

            Build couchbase-server-7.1.0-1424 contains backup commit 5b3ecaf with commit message:
            MB-48744 Don't backup/restore views for unsupported configurations

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1424 contains backup commit 5b3ecaf with commit message: MB-48744 Don't backup/restore views for unsupported configurations

            I have a good run on 7.1.0-1424. I close this ticket.

            http://perf.jenkins.couchbase.com/job/rhea-5node2/1493/ 

            bo-chun.wang Bo-Chun Wang added a comment - I have a good run on 7.1.0-1424. I close this ticket. http://perf.jenkins.couchbase.com/job/rhea-5node2/1493/  

            People

              bo-chun.wang Bo-Chun Wang
              bo-chun.wang Bo-Chun Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty