Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46013

[Windows] cbbackupmgr failed due to "Stream request for vBucket 365 timed out after 3m0s"

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • tools
    • Enterprise Edition 7.0.0 build 5060

    Description

      Build: 7.0.0 - 5060

      Steps:

      • Cluster with index, cbas, fts and eventing services actively working on bucket data
      • Constantly running rebalances (in/out/swap) of kv / n1ql:index / cbas nodes
      • Scheduled backup on full cluster and only_services as two tasks.

      Observing few cluster backup tasks failed due to reason "Stream request for vBucket 365 timed out after 3m0s"

      "node_runs": [
          {
            "node_id": "2989c672ce3225a540a566ef716812d6",
            "status": "failed",
            "start": "2021-04-29T17:00:18.0714657-07:00",
            "end": "2021-04-29T17:04:00.6665167-07:00",
            "error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'travel-sample': failed to transfer bucket data for bucket 'travel-sample': failed to transfer key value data: failed to transfer key value data: failed to open stream: failed to stream vBucket 365: client received unexpected error 'Stream request for vBucket 365 timed out after 3m0s'",
            "progress": 0,
            "stats": {
              "error": "failed to execute cluster operations: failed to execute bucket operation for bucket 'travel-sample': failed to transfer bucket data for bucket 'travel-sample': failed to transfer key value data: failed to transfer key value data: failed to open stream: failed to stream vBucket 365: client received unexpected error 'Stream request for vBucket 365 timed out after 3m0s'",
              "stats": {
                "started_at": 1619740818323472100,
                "buckets": {
                  "travel-sample": {
                    "estimated_total_items": 196499167,
                    "total_items": 17304970,
                    "total_vbuckets": 1024,
                    "vbuckets_complete": 915,
                    "bytes_received": 843303955,
                    "snapshot_markers_received": 915,
                    "snapshot_ends_received": 915,
                    "failover_logs_received": 915,
                    "mutations_received": 2306919,
                    "deletions_received": 5403716,
                    "started_at": 1619740852033783600,
                    "complete": false,
                    "errored": false
                  }
                },
                "complete": false
              }
            },
            "error_code": 2
          }
        ],
            

      Note: Cluster has only one backup node (172.23.136.115)

      Logs from backup folder: https://cb-engineering.s3.amazonaws.com/cbbackup_failure/backup_logs.tar.gz

      Attachments

        1. backup_failures.png
          backup_failures.png
          497 kB
        2. backup_repos.png
          backup_repos.png
          550 kB
        3. backupmgr_error.png
          backupmgr_error.png
          559 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            ashwin.govindarajulu Ashwin Govindarajulu created issue -
            carlos.gonzalez Carlos Gonzalez Betancort (Inactive) made changes -
            Field Original Value New Value
            Link This issue relates to MB-46003 [ MB-46003 ]
            carlos.gonzalez Carlos Gonzalez Betancort (Inactive) made changes -
            Assignee Daniel Owen [ owend ] Carlos Gonzalez Betancort [ carlos.gonzalez ]
            owend Daniel Owen made changes -
            Sprint Tools: CC Final Sprint [ 1549 ]
            drigby Dave Rigby made changes -
            Component/s couchbase-bucket [ 10173 ]
            james.lee James Lee made changes -
            Assignee Carlos Gonzalez Betancort [ carlos.gonzalez ] James Lee [ james.lee ]
            james.lee James Lee made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            owend Daniel Owen made changes -
            Due Date 04/May/21
            owend Daniel Owen made changes -
            Due Date 04/May/21 06/May/21
            james.lee James Lee made changes -
            Assignee James Lee [ james.lee ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            joe.mitchelljones Joe Mitchell Jones made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.0 [ 17233 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s Cheshire-Cat [ 15915 ]

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ashwin.govindarajulu Ashwin Govindarajulu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty