Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-35905

cbbackupmgr and cbexport hang when running with error `get vBucket item count failed for vBucket 0, operation timed out after (20 seconds)

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • Yes

    Description

      Install Couchbase server 6.5.0-4218 on 4 centos 7.6 servers.
      Create a 3 nodes cluster (224, 225 and 226) with services: kv,index,n1ql,fts
      Create default bucket and load 1000 docs to default bucket.
      Update and delete data in default bucket so that it goes down to 600 docs left in bucket.
      On node 172.23.106.210, run backup with cbbackupmgr, backup hang forever
      Check backup logs, see a lot of errors:

      [root@localhost ~]# tail -f /tmp/entbackup/logs/backup-0.log 
      2019-09-10T12:31:03.282-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      2019-09-10T12:31:03.283-07:00 (Plan) Deciding which key value data to transfer for default
      2019-09-10T12:31:03.286-07:00 (Rest) GET http://172.23.121.224:8091/pools/default/nodeServices 200
      2019-09-10 19:31:03.416527808 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:31:08.300-07:00 WARN: Received error `get vBucket item count failed for vBucket 0, operation timed out after (5 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:31:08.404650254 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:31:18.300-07:00 WARN: Received error `get vBucket item count failed for vBucket 0, operation timed out after (10 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:31:18.406742421 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:31:33.301-07:00 WARN: Received error `get vBucket item count failed for vBucket 0, operation timed out after (15 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:31:33.405238548 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:31:53.301-07:00 WARN: Received error `get vBucket item count failed for vBucket 0, operation timed out after (20 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:31:53.405054617 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:32:18.302-07:00 WARN: Received error `get vBucket item count failed for vBucket 0, operation timed out after (25 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:32:18.407259926 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:32:23.302-07:00 WARN: Received error `get vBucket item count failed for vBucket 1, operation timed out after (5 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:32:23.409981178 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:32:33.302-07:00 WARN: Received error `get vBucket item count failed for vBucket 1, operation timed out after (10 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:32:33.40738486 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:32:48.303-07:00 WARN: Received error `get vBucket item count failed for vBucket 1, operation timed out after (15 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:32:48.407415712 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      2019-09-10T12:33:08.303-07:00 WARN: Received error `get vBucket item count failed for vBucket 1, operation timed out after (20 seconds)` while trying to get vbucket item count, will retry -- couchbase.(*dcpClient).GetItemsRemaining() at dcp.go:645
      2019-09-10 19:33:08.407561501 +0000 UTC (Gocbcore) Attempted to start tracing on traced request
      ^Z
      [1]+  Stopped                 tail -f /tmp/entbackup/logs/backup-0.log
      

      It looks like same issue with bug MB-35588 with different errors.
      I will run again with latest build to see if I could reproduce this issue.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              james.lee James Lee
              thuan Thuan Nguyen
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty