Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.1.2
-
Untriaged
-
1
-
No
Description
7.1.2-3345
Test:
-test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_couchstore.yml
Scale 3
Iteration 2
Task failed with error:
{
|
"task_name": "backup-1",
|
"status": "failed",
|
"start": "2022-07-07T16:00:16.202729516-07:00",
|
"end": "2022-07-07T18:05:14.281993182-07:00",
|
"node_runs": [
|
{
|
"node_id": "87cd53a1c81339b58356509559c450a9",
|
"status": "failed",
|
"start": "2022-07-07T16:25:55.243817576-07:00",
|
"end": "2022-07-07T18:05:14.082624421-07:00",
|
"error": "exit status 1",
|
"progress": 80.0636574074074,
|
"stats": {
|
"id": "48518622-3867-4890-90bd-b629cf546a86",
|
"current_transfer": 1,
|
"total_transfers": 1,
|
"transfers": [
|
{
|
"description": "Backing up to 2022-07-07T16_46_20.890216605-07_00",
|
"stats": {
|
"started_at": 1657236381998106000,
|
"buckets": {
|
"bucket1": {
|
"estimated_total_items": 137977210,
|
"total_items": 3365176,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 1936702992,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"mutations_received": 1650262,
|
"deletions_received": 1716645,
|
"started_at": 1657240391847722000,
|
"finished_at": 1657240828665237200,
|
"complete": true
|
},
|
"bucket2": {
|
"estimated_total_items": 206412276,
|
"total_items": 8103088,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 497928695,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"deletions_received": 8103088,
|
"started_at": 1657241900306638000,
|
"finished_at": 1657241986586089700,
|
"complete": true
|
},
|
"bucket3": {
|
"estimated_total_items": 266684277,
|
"total_items": 6979973,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 660417560,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"mutations_received": 46224,
|
"deletions_received": 6933749,
|
"started_at": 1657241100464234800,
|
"finished_at": 1657241159390943700,
|
"complete": true
|
},
|
"bucket4": {
|
"estimated_total_items": 47978775,
|
"total_vbuckets": 1024,
|
"started_at": 1657242306529310200
|
},
|
"bucket5": {
|
"estimated_total_items": 106476573,
|
"total_items": 651000,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 390976685,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"mutations_received": 488250,
|
"deletions_received": 162750,
|
"started_at": 1657241565543724300,
|
"finished_at": 1657241665009197300,
|
"complete": true
|
},
|
"bucket7": {
|
"estimated_total_items": 199485501,
|
"total_items": 735420,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 465521854,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"mutations_received": 611070,
|
"deletions_received": 122318,
|
"started_at": 1657237587048243200,
|
"finished_at": 1657237724435557400,
|
"complete": true
|
},
|
"bucket8": {
|
"estimated_total_items": 565460,
|
"total_items": 27370,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 82592,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"started_at": 1657237954550507000,
|
"finished_at": 1657237956344404200,
|
"complete": true
|
},
|
"bucket9": {
|
"estimated_total_items": 573326,
|
"total_items": 25214,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 82640,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"started_at": 1657237959709916700,
|
"finished_at": 1657237961283413000,
|
"complete": true
|
},
|
"default": {
|
"estimated_total_items": 617379672,
|
"total_items": 13645757,
|
"total_vbuckets": 1024,
|
"vbuckets_complete": 1024,
|
"bytes_received": 12400356519,
|
"snapshot_markers_received": 1024,
|
"failover_logs_received": 1024,
|
"mutations_received": 2817830,
|
"deletions_received": 10382576,
|
"started_at": 1657237965361846000,
|
"finished_at": 1657239962489315600,
|
"complete": true
|
}
|
}
|
},
|
"progress": 80.0636574074074,
|
"eta": "2022-07-07T18:24:51.519326642-07:00"
|
}
|
],
|
"progress": 80.0636574074074,
|
"eta": "2022-07-07T18:24:51.519326642-07:00"
|
},
|
"error_code": 2
|
}
|
],
|
"error": "exit status 1",
|
"error_code": 2,
|
"type": "BACKUP",
|
"show": true
|
}
|
From CBM logs:
2022-07-07T18:05:06.540-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:06.556-07:00 (Gocbcore) Pipeline Client 0xc21ddfe660 failed to bootstrap: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"8a3a4d63-af5d-498b-df53-f3f8e1663039","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49160","last_connection_id":"e3aa8615606b1f9d/44eb126b768f4bbc"}
|
2022-07-07T18:05:06.999-07:00 (Couchbase) Successfully created DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:06.999-07:00 (Couchbase) Creating DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.011-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:07.030-07:00 (Gocbcore) Pipeline Client 0xc21b418840 failed to bootstrap: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"68b5046a-85de-4e78-8df2-b3ab3b1150ef","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49194","last_connection_id":"d5e737e4a0a17088/ca3e1f983bbd06fd"}
|
2022-07-07T18:05:07.067-07:00 (Couchbase) Successfully created DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.067-07:00 (Couchbase) Creating DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.117-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:07.143-07:00 (Gocbcore) Pipeline Client 0xc00f820a80 failed to bootstrap: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"882d56c2-2d4b-4f0f-55c6-b8c9e01e7239","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49212","last_connection_id":"3a241efb4920d17e/e85617d88a59f8fa"}
|
2022-07-07T18:05:07.242-07:00 (Couchbase) Successfully created DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.242-07:00 (Couchbase) Creating DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.260-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:07.314-07:00 (Couchbase) Successfully created DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.314-07:00 (Couchbase) Creating DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.350-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:07.365-07:00 (Gocbcore) Pipeline Client 0xc081f757a0 failed to bootstrap: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"f78f59d1-402b-4ee1-c6fe-89b106d7cedd","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49260","last_connection_id":"e70862db840e3225/0b5fa59eff73b0f4"}
|
2022-07-07T18:05:07.388-07:00 (Couchbase) Successfully created DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.388-07:00 (Couchbase) Creating DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.421-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:07.440-07:00 (Gocbcore) Pipeline Client 0xc00103e120 failed to bootstrap: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"af5e5aa9-6c32-425c-d042-bc6ee58dded5","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49290","last_connection_id":"ee3817232419c90c/8e954f6ff569c57f"}
|
2022-07-07T18:05:07.855-07:00 (Couchbase) Successfully created DCP connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.855-07:00 (Couchbase) Creating memcached connection to cluster 'couchbase://172.23.105.107:11210,172.23.106.100:11210,172.23.108.103:11210,172.23.121.117:11210,172.23.123.28:11210,172.23.97.119:11210,172.23.97.121:11210,172.23.97.122:11210,172.23.99.20:11210,172.23.99.21:11210,172.23.99.25:11210'
|
2022-07-07T18:05:07.873-07:00 WARN: (Auth) (Service) Could not get credentials for host '172.23.106.100:11210': rpc error: code = Internal desc = failed to get credentials from cbauth: Unable to find given hostport in cbauth database: `172.23.106.100:11210' -- auth.(*BaseProvider).GetCredentials() at base_provider.go:101
|
2022-07-07T18:05:07.880-07:00 (Gocbcore) Pipeline Client 0xc0010817a0 failed to bootstrap: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"2c765eb1-a2ba-4a09-87dd-d5c5409d21c6","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49320","last_connection_id":"adbd313d4625e180/9a9cf90b5a285b4a"}
|
2022-07-07T18:05:09.124-07:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket4': failed to transfer bucket data for bucket 'bucket4': failed to transfer key value data: failed to transfer key value data: failed to get bucket item count: failed to create gocbcore agent: agent failed to connect to the cluster: authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"2c765eb1-a2ba-4a09-87dd-d5c5409d21c6","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49320","last_connection_id":"adbd313d4625e180/9a9cf90b5a285b4a"}
|
2022-07-07T18:05:10.857-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. authentication failure | {"status_code":32,"document_key":"SCRAM-SHA512","bucket":"bucket4","error_name":"AUTH_ERROR","error_description":"Authentication failed","opaque":5,"ref":"2c765eb1-a2ba-4a09-87dd-d5c5409d21c6","last_dispatched_to":"172.23.106.100:11210","last_dispatched_from":"172.23.104.69:49320","last_connection_id":"adbd313d4625e180/9a9cf90b5a285b4a"}
|
Attaching CBM logs
Server logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.104.137.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.104.155.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.104.157.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.104.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.104.67.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.104.69.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.105.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.105.111.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.105.168.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.106.100.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.106.188.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.108.103.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.120.107.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.120.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.121.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.123.28.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.96.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.96.192.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.96.251.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.96.252.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.96.253.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.97.121.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.97.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.97.239.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.99.11.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.99.21.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1657244922/collectinfo-2022-07-08T014848-ns_1%40172.23.99.25.zip
Attachments
Issue Links
- depends on
-
-
- Resolved
-
- relates to
-
GOCBC-1324 WaitUntilReady should check all pipelines are errored when no config seen
-
- Open
-