Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-39632

[CBM] cbbackupmgr failed to run backup if memcached restarted

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 1
    • Yes

    Description

      Install Couchbase server 6.6.0-7705 on 9 centos servers

      Create a bucket default and load 50K docs to bucket.

      Do multi backup and merge.  During backup, we do pause memcached process (

      killall -SIGSTOP memcached) in 8 seconds and release it (

      killall -SIGCONT memcached).

      There is one run that cbbackupmgr failed to backup with timeout error

      2020-05-28T16:35:01.474-07:00 (Cmd) backup --archive /tmp/entbackup_172.23.121.224 --repo backup --cluster http://172.23.121.224:8091 --username <ud>Administrator</ud> --password ******** --no-progress-bar 
      2020-05-28T16:35:01.474-07:00 (Cmd) mounted archive with id: 84017afb-1995-47e1-941f-700b79e71190
      		 
      2020-05-28T16:35:01.481-07:00 (Rest) GET http://172.23.121.224:8091/pools 200
      		 
      2020-05-28T16:35:01.497-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets 200
      		 
      2020-05-28T16:35:01.509-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets/default 200
      		 
      2020-05-28T16:35:01.512-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets/default/collections 400
      		 
      2020-05-28T16:35:01.512-07:00 WARN: (Rest) this version of Couchbase Server does not support collections -- couchbase.(*RestClient).GetCollectionManifest() at rest.go:230
      		 
      2020-05-28T16:35:01.525-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets 200
      		 
      2020-05-28T16:35:01.537-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets/default 200
      		 
      2020-05-28T16:35:01.544-07:00 (Rest) GET http://172.23.121.224:8091/pools 200
      		 
      2020-05-28T16:35:01.544-07:00 (Cmd) Backing up cluster 7055cbd0113aa45ff1e2ddb97e6da51f
      		 
      2020-05-28T16:35:01.548-07:00 (Stats) Starting stat gathering - stat timestamp: 1590708901
      		 
      2020-05-28T16:35:01.548-07:00 (Plan) Transferring cluster configuration
      		 
      2020-05-28T16:35:01.551-07:00 (Rest) GET http://172.23.121.224:8091/pools 200
      		 
      2020-05-28T16:35:01.551-07:00 (Plan) Successfully transferred cluster config
      		 
      2020-05-28T16:35:01.552-07:00 (Plan) Transferring Eventing metadata
      		 
      2020-05-28T16:35:01.554-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:01.555-07:00 (Plan) Successfully transferred Eventing metadata
      		 
      2020-05-28T16:35:01.555-07:00 (Plan) Transferring full text index aliases
      		 
      2020-05-28T16:35:01.557-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:01.562-07:00 (Rest) GET http://172.23.121.224:8094/api/index 200
      		 
      2020-05-28T16:35:01.562-07:00 (Plan) Successfully transferred full text index aliases
      		 
      2020-05-28T16:35:01.563-07:00 (Plan) Transferring bucket configuration for 'default' to 'default'
      		 
      2020-05-28T16:35:01.573-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets 200
      		 
      2020-05-28T16:35:01.575-07:00 (Plan) Successfully transferred bucket configuration for 'default' to 'default'
      		 
      2020-05-28T16:35:01.576-07:00 (Plan) Transferring Analytics metadata
      		 
      2020-05-28T16:35:01.578-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:01.578-07:00 (Plan) Successfully transferred Analytics metadata
      		 
      2020-05-28T16:35:01.579-07:00 (Plan) Transferring full text index definitions for 'default' to 'default'
      		 
      2020-05-28T16:35:01.581-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:01.584-07:00 (Rest) GET http://172.23.121.224:8094/api/index 200
      		 
      2020-05-28T16:35:01.585-07:00 (Plan) Successfully transferred full text index definitions for 'default' to 'default'
      		 
      2020-05-28T16:35:01.585-07:00 (Plan) Transferring GSI index definitions for default to default
      		 
      2020-05-28T16:35:01.587-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:01.627-07:00 (Rest) GET http://172.23.121.224:9102/getIndexMetadata?bucket=default 200
      		 
      2020-05-28T16:35:01.628-07:00 (Plan) Successfully transferred GSI index definitions for default to default
      		 
      2020-05-28T16:35:01.629-07:00 (Plan) Transferring views definitions from 'default' to 'default'
      		 
      2020-05-28T16:35:01.640-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets 200
      		 
      2020-05-28T16:35:01.649-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:01.653-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets/default/ddocs 200
      		 
      2020-05-28T16:35:01.654-07:00 (Plan) Successfully transferred views definitions from 'default' to 'default'
      		 
      2020-05-28T16:35:01.655-07:00 (Plan) Checking for data movement restrictions between default and default
      		 
      2020-05-28T16:35:01.665-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/buckets 200
      		 
      2020-05-28T16:35:01.673-07:00 (Rest) GET http://172.23.121.224:8091/pools/default 200
      		 
      2020-05-28T16:35:01.674-07:00 Transferring from Couchbase Server 6.6.0
      		 
      2020-05-28T16:35:01.675-07:00 (Plan) Successfully checked for data movement restrictions between default and default
      		 
      2020-05-28T16:35:01.675-07:00 (Plan) Deciding which key value data to transfer for default
      		 
      2020-05-28T16:35:01.677-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      		 
      2020-05-28T16:35:33.483-07:00 (Stats) Stopping stat collection
      		 
      2020-05-28T16:35:33.487-07:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operations: failed to transfer bucket data for bucket 'default': failed to transfer data range: failed to determine which data to transfer for bucket 'default': failed to get gocbcore DCP agent: agent failed to connect to the cluster: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":30000183963,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2020-05-28T16:38:09.481-07:00 Error reading range.json: stat /tmp/entbackup_172.23.121.224/backup/2020-05-28T16_35_01.514050369-07_00/default-33a7efcf72ffe319f57260dad05ca748/range.json: no such file or directory
      		 
      2020-05-28T16:38:09.482-07:00 Error getting last persisted seqnos: Backup Meta `/tmp/entbackup_172.23.121.224/backup/2020-05-28T16_35_01.514050369-07_00/default-33a7efcf72ffe319f57260dad05ca748/range.json` not found
      		 
      2020-05-28T17:46:27.904-07:00 Error reading range.json: stat /tmp/entbackup_172.23.121.224/backup/2020-05-28T16_35_01.514050369-07_00/default-33a7efcf72ffe319f57260dad05ca748/range.json: no such file or directory
      		 
      2020-05-28T17:46:27.905-07:00 Error getting last persisted seqnos: Backup Meta `/tmp/entbackup_172.23.121.224/backup/2020-05-28T16_35_01.514050369-07_00/default-33a7efcf72ffe319f57260dad05ca748/range.json` not found
      		 
      [root@localhost ~]#

       

       

       

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. GOCBC-908 'WaitUntilReady' appears to have inconsistent retry behviour

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thuan Thuan Nguyen
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty