Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
7.1.0
-
Untriaged
-
1
-
Unknown
-
Tools 2021 Nov
Description
7.1.0-1461
Test:
-test tests/integration/neo/test_neo_magma_wo_gsi_n1ql.yml -scope tests/integration/neo/scope_neo_magma_wo_gsi_n1ql.yml
Scale 2
Iteration 2
Backup task hung for several hours
From CBM logs:
2021-10-11T04:02:00.484-07:00 (DCP) (bucket7) (vb 801) Stream closed because all items were streamed | {"uuid":62798241640176,"snap_start":0,"snap_end":36358,"last_seqno":36358,"retries":0}
|
2021-10-11T04:02:00.806-07:00 (DCP) (bucket7) (vb 797) Stream closed because all items were streamed | {"uuid":253388666118623,"snap_start":0,"snap_end":31822,"last_seqno":31822,"retries":0}
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 436) Stream has been inactive for 1m0s, last seqno 6956 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 799) Stream has been inactive for 1m0s, last seqno 6986 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 722) Stream has been inactive for 1m0s, last seqno 13196 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 583) Stream has been inactive for 1m0s, last seqno 13447 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 546) Stream has been inactive for 1m0s, last seqno 13332 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 796) Stream has been inactive for 1m0s, last seqno 13404 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 755) Stream has been inactive for 1m0s, last seqno 7470 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 840) Stream has been inactive for 1m0s, last seqno 13753 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 562) Stream has been inactive for 1m0s, last seqno 13448 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
|
2021-10-11T04:58:37.645-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T06:15:04.694-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T07:46:43.017-07:00 (Gocbcore) memdClient read failure on conn `4e9059f78b617115/d2d1bd465a7c8c6e` : EOF
|
2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `c7732dd77ba2b0bc/2f9f6dd0545f87b9` : EOF
|
2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `73b0f00f10ffc6f4/e6444db82a73ada4` : EOF
|
2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `b6f80ccb11f55f00/418c63d17674d064` : EOF
|
2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `1b83a94f7922c11b/c0fa1a809178da27` : EOF
|
2021-10-11T07:46:43.032-07:00 (Gocbcore) Pipeline Client 0xc00047c690 failed to bootstrap: bucket not found
|
2021-10-11T07:46:43.032-07:00 (Gocbcore) Pipeline Client 0xc00047c070 failed to bootstrap: bucket not found
|
2021-10-11T07:46:43.033-07:00 (Gocbcore) Pipeline Client 0xc00047c000 failed to bootstrap: bucket not found
|
2021-10-11T07:46:43.033-07:00 (Gocbcore) Pipeline Client 0xc00047c1c0 failed to bootstrap: bucket not found
|
2021-10-11T07:46:43.035-07:00 (Gocbcore) Pipeline Client 0xc00047c4d0 failed to bootstrap: bucket not found
|
2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `73b0f00f10ffc6f4/1dda97c7d8244ab4` : EOF
|
2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `4e9059f78b617115/04926a9743d0bcc7` : EOF
|
2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `b6f80ccb11f55f00/52460fbb4c05cde0` : EOF
|
2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `5412ba1dbd4fae57/f263ebf826e134c4` : EOF
|
2021-10-11T08:18:48.169-07:00 (Gocbcore) Pipeline Client 0xc0005f10a0 failed to bootstrap: bucket not found
|
2021-10-11T08:18:48.169-07:00 (Gocbcore) Pipeline Client 0xc06d3fae00 failed to bootstrap: bucket not found
|
2021-10-11T08:18:48.169-07:00 (Gocbcore) Pipeline Client 0xc06d3faee0 failed to bootstrap: bucket not found
|
2021-10-11T08:18:48.173-07:00 (Gocbcore) Pipeline Client 0xc0005f0cb0 failed to bootstrap: bucket not found
|
2021-10-11T09:45:25.724-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:45:25.724-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:45:36.811-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"3e54d0c6-c295-4548-9839-eb696d8ced53","TimeObserved":5001213766,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.121.77:8091","LastDispatchedFrom":"","LastConnectionID":""}
|
2021-10-11T09:45:36.811-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"f35ca1e4-4181-4f5e-af0a-b858951b65d9","TimeObserved":5000322598,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.121.77:8091","LastDispatchedFrom":"","LastConnectionID":""}
|
2021-10-11T09:45:41.812-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"ed42ccc7-d93a-4312-87c7-ca23f1f60d34","TimeObserved":5000676961,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.123.24:8091","LastDispatchedFrom":"","LastConnectionID":""}
|
2021-10-11T09:45:41.812-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"8fc2ed8b-94df-45c6-a4ce-61517d56c0d9","TimeObserved":5000664540,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.123.24:8091","LastDispatchedFrom":"","LastConnectionID":""}
|
2021-10-11T09:45:48.771-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:45:51.771-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:46:59.271-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:47:51.308-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:48:14.329-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:48:59.867-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T09:55:59.961-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
2021-10-11T10:02:55.236-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:03:13.875-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:03:13.876-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:04:35.312-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:04:53.919-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:04:53.919-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:06:15.360-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:06:33.962-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T10:06:33.962-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
|
2021-10-11T11:12:28.902-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.97.74.zip
Attaching backup logs
Attachments
Issue Links
- relates to
-
MB-49331 [CBM] Authentication error when backing up during a rebalance
- Resolved
-
MB-49332 [CBM] EOF when backing up during a rebalance
- Resolved
-
MB-49334 [CBM] 'connection refused' when backing up during a rebalance
- Resolved
-
MB-49374 [System Test][Backup Service] backup tasks failed with error - failed to process snapshot marker: received index create for vBucket 922 before opening
- Closed
-
MB-49375 [System Test][Backup Service] backup tasks failed with error - timeout reached whilst waiting for manifest id 123 to have propagated throughout the cluster, waited 1m0s
- Closed