Details
-
Bug
-
Resolution: Fixed
-
Major
-
7.1.0
-
Untriaged
-
1
-
No
-
Tools 2021 Dec, Tools 2022-Jan
Description
7.1.0-1912
Test:
-test tests/integration/neo/test_neo_magma_milestone4.yml -scope tests/integration/neo/scope_neo_magma.yml
backup task failed with error:
{
|
"task_name": "backup-1",
|
"status": "failed",
|
"start": "2021-12-16T02:01:03.312008297-08:00",
|
"end": "2021-12-16T02:02:32.149821903-08:00",
|
"node_runs": [
|
{
|
"node_id": "eebd58d7bbb5aaf005bce2dfb3a26c69",
|
"status": "failed",
|
"start": "2021-12-16T02:01:03.39611296-08:00",
|
"end": "2021-12-16T02:02:32.072185092-08:00",
|
"error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: EOF",
|
"progress": 0,
|
"stats": {
|
"error": "failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: EOF",
|
"stats": {
|
"started_at": 1639648864236425200,
|
"buckets": {
|
"NEW_ORDER": {
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 33136,
|
"failover_logs_received": 1024,
|
"started_at": 1639648869561016600,
|
"finished_at": 1639648871341710000,
|
"complete": true
|
},
|
"bucket4": {
|
"estimated_total_items": 94125813,
|
"total_items": 6486719,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 5129443476,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"mutations_received": 6486719,
|
"deletions_received": 479774,
|
"started_at": 1639648875051976700,
|
"finished_at": 1639648913860701700,
|
"complete": true
|
},
|
"bucket6": {
|
"estimated_total_items": 16566791,
|
"total_items": 14342987,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 271,
|
"bytes_received": 4573748812,
|
"snapshot_markers_received": 978,
|
"failover_logs_received": 1024,
|
"mutations_received": 5810112,
|
"deletions_received": 89528,
|
"started_at": 1639648918825120500
|
}
|
}
|
}
|
},
|
"error_code": 2
|
}
|
],
|
"error": "exit status 1: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: EOF",
|
"error_code": 2,
|
"type": "BACKUP",
|
"show": true
|
}
|
From CBM logs:
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 261) Received an unexpected error whilst streaming, beginning teardown: EOF -- couchbase.(*DCPAsyncWorker).handleDCPError() at dcp_async_worker.go:615
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 148) Stream closed by backup client | {"uuid":78241516948860,"snap_start":0,"snap_end":16555,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 113) Stream closed by backup client | {"uuid":41675207136100,"snap_start":0,"snap_end":36,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 101) Stream closed by backup client | {"uuid":213530439858746,"snap_start":0,"snap_end":16019,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 189) Stream closed by backup client | {"uuid":152122049309803,"snap_start":0,"snap_end":16279,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 142) Stream closed by backup client | {"uuid":20885656787878,"snap_start":0,"snap_end":16415,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 139) Stream closed by backup client | {"uuid":117798221016366,"snap_start":0,"snap_end":16490,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 131) Stream closed by backup client | {"uuid":94281051914404,"snap_start":0,"snap_end":16490,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 110) Stream closed by backup client | {"uuid":161758497046964,"snap_start":0,"snap_end":16868,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.575-08:00 WARN: (DCP) (bucket6) (vb 106) Stream closed by backup client | {"uuid":187672254614228,"snap_start":0,"snap_end":15821,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:535
|
2021-12-16T02:02:29.579-08:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket6': failed to transfer bucket data for bucket 'bucket6': failed to transfer key value data: failed to transfer key value data: EOF
|
2021-12-16T09:00:06.742-08:00 (Cmd) cbbackupmgr version 7.1.0-1912 Hostname: localhost.localdomain OS: linux Version: 3.10.0-1160.49.1.el7.x86_64 Arch: amd64 vCPU: 8 Memory: 25083994112 (23.36GiB)
|
2021-12-16T09:00:06.742-08:00 (Cmd) backup --cluster http://172.23.108.144:8091 --archive /data/archive --repo 131358af-1241-4a13-a538-8c8f9817f038 --purge --auto-select-threads --json-progress 3d735c4f-5bd9-44b3-a4b2-60a6f7ad5112 --full-backup
|
2021-12-16T09:00:06.742-08:00 (Cmd) mounted archive with id: e0654604-225f-46a7-8fd9-ed27364d0202
|
During this time, the test was doing a rebalance:
[2021-12-15T01:46:35-08:00, sequoiatools/couchbase-cli:7.1:00a4db] rebalance -c 172.23.108.139:8091 --server-remove 172.23.108.140 -u Administrator -p password
|
[2021-12-15T02:18:58-08:00, sequoiatools/cmd:297981] 60
|
Cluster config:
########## Cluster config ##################
|
###### backup : 1 ===== > [172.23.108.144:8091] ###########
|
###### index : 6 ===== > [172.23.104.249:8091 172.23.105.0:8091 172.23.105.39:8091 172.23.106.54:8091 172.23.108.132:8091 172.23.108.140:8091] ###########
|
###### n1ql : 2 ===== > [172.23.106.37:8091 172.23.108.146:8091] ###########
|
###### kv : 11 ===== > [172.23.107.236:8091 172.23.107.25:8091 172.23.108.134:8091 172.23.108.136:8091 172.23.108.138:8091 172.23.108.139:8091 172.23.108.141:8091 172.23.108.142:8091 172.23.108.143:8091 172.23.108.145:8091 172.23.108.148:8091] ###########
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.104.249.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.105.0.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.105.38.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.105.39.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.106.37.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.107.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.107.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.132.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.138.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.139.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.140.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.141.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.142.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.143.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.144.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.145.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.146.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1639651480/collectinfo-2021-12-16T104442-ns_1%40172.23.108.148.zip
Attachments
Issue Links
- relates to
-
MB-50171 Data service chould return a more informative error for the closed backup DCP streams
- Resolved