Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48856

[System Test][Backup Service] task hung for several hours

    XMLWordPrintable

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Critical
    • 7.1.0
    • 7.1.0
    • tools
    • Untriaged
    • 1
    • Unknown
    • Tools 2021 Nov

    Description

      7.1.0-1461

      Test:
      -test tests/integration/neo/test_neo_magma_wo_gsi_n1ql.yml -scope tests/integration/neo/scope_neo_magma_wo_gsi_n1ql.yml
      Scale 2
      Iteration 2

      Backup task hung for several hours

      From CBM logs:

      2021-10-11T04:02:00.484-07:00 (DCP) (bucket7) (vb 801) Stream closed because all items were streamed | {"uuid":62798241640176,"snap_start":0,"snap_end":36358,"last_seqno":36358,"retries":0}
      		 
      2021-10-11T04:02:00.806-07:00 (DCP) (bucket7) (vb 797) Stream closed because all items were streamed | {"uuid":253388666118623,"snap_start":0,"snap_end":31822,"last_seqno":31822,"retries":0}
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 436) Stream has been inactive for 1m0s, last seqno 6956 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 799) Stream has been inactive for 1m0s, last seqno 6986 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 722) Stream has been inactive for 1m0s, last seqno 13196 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 583) Stream has been inactive for 1m0s, last seqno 13447 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 546) Stream has been inactive for 1m0s, last seqno 13332 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 796) Stream has been inactive for 1m0s, last seqno 13404 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 755) Stream has been inactive for 1m0s, last seqno 7470 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 840) Stream has been inactive for 1m0s, last seqno 13753 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:03:09.274-07:00 WARN: (DCP) (bucket7) (vb 562) Stream has been inactive for 1m0s, last seqno 13448 -- couchbase.(*DCPAsyncWorker).monitorFunc.func1() at dcp_async_worker.go:247
      		 
      2021-10-11T04:58:37.645-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T06:15:04.694-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T07:46:43.017-07:00 (Gocbcore) memdClient read failure on conn `4e9059f78b617115/d2d1bd465a7c8c6e` : EOF
      		 
      2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `c7732dd77ba2b0bc/2f9f6dd0545f87b9` : EOF
      		 
      2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `73b0f00f10ffc6f4/e6444db82a73ada4` : EOF
      		 
      2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `b6f80ccb11f55f00/418c63d17674d064` : EOF
      		 
      2021-10-11T07:46:43.018-07:00 (Gocbcore) memdClient read failure on conn `1b83a94f7922c11b/c0fa1a809178da27` : EOF
      		 
      2021-10-11T07:46:43.032-07:00 (Gocbcore) Pipeline Client 0xc00047c690 failed to bootstrap: bucket not found
      		 
      2021-10-11T07:46:43.032-07:00 (Gocbcore) Pipeline Client 0xc00047c070 failed to bootstrap: bucket not found
      		 
      2021-10-11T07:46:43.033-07:00 (Gocbcore) Pipeline Client 0xc00047c000 failed to bootstrap: bucket not found
      		 
      2021-10-11T07:46:43.033-07:00 (Gocbcore) Pipeline Client 0xc00047c1c0 failed to bootstrap: bucket not found
      		 
      2021-10-11T07:46:43.035-07:00 (Gocbcore) Pipeline Client 0xc00047c4d0 failed to bootstrap: bucket not found
      		 
      2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `73b0f00f10ffc6f4/1dda97c7d8244ab4` : EOF
      		 
      2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `4e9059f78b617115/04926a9743d0bcc7` : EOF
      		 
      2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `b6f80ccb11f55f00/52460fbb4c05cde0` : EOF
      		 
      2021-10-11T08:18:48.137-07:00 (Gocbcore) memdClient read failure on conn `5412ba1dbd4fae57/f263ebf826e134c4` : EOF
      		 
      2021-10-11T08:18:48.169-07:00 (Gocbcore) Pipeline Client 0xc0005f10a0 failed to bootstrap: bucket not found
      		 
      2021-10-11T08:18:48.169-07:00 (Gocbcore) Pipeline Client 0xc06d3fae00 failed to bootstrap: bucket not found
      		 
      2021-10-11T08:18:48.169-07:00 (Gocbcore) Pipeline Client 0xc06d3faee0 failed to bootstrap: bucket not found
      		 
      2021-10-11T08:18:48.173-07:00 (Gocbcore) Pipeline Client 0xc0005f0cb0 failed to bootstrap: bucket not found
      		 
      2021-10-11T09:45:25.724-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:45:25.724-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:45:36.811-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"3e54d0c6-c295-4548-9839-eb696d8ced53","TimeObserved":5001213766,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.121.77:8091","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2021-10-11T09:45:36.811-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"f35ca1e4-4181-4f5e-af0a-b858951b65d9","TimeObserved":5000322598,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.121.77:8091","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2021-10-11T09:45:41.812-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"ed42ccc7-d93a-4312-87c7-ca23f1f60d34","TimeObserved":5000676961,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.123.24:8091","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2021-10-11T09:45:41.812-07:00 (Gocbcore) Failed to connect to host. ambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"ambiguous timeout"}},"OperationID":"http","Opaque":"8fc2ed8b-94df-45c6-a4ce-61517d56c0d9","TimeObserved":5000664540,"RetryReasons":null,"RetryAttempts":0,"LastDispatchedTo":"http://172.23.123.24:8091","LastDispatchedFrom":"","LastConnectionID":""}
      		 
      2021-10-11T09:45:48.771-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:45:51.771-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:46:59.271-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:47:51.308-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:48:14.329-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:48:59.867-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T09:55:59.961-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout
      		 
      2021-10-11T10:02:55.236-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:03:13.875-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:03:13.876-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:04:35.312-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:04:53.919-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:04:53.919-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:06:15.360-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:06:33.962-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T10:06:33.962-07:00 (Gocbcore) Failed to connect to host. Get "http://172.23.96.14:8091/pools/default/bs/bucket7": dial tcp 172.23.96.14:8091: connect: connection refused
      		 
      2021-10-11T11:12:28.902-07:00 (Gocbcore) CCCPPOLL: Failed to retrieve CCCP config. ambiguous timeout

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.120.74.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1633954267/collectinfo-2021-10-11T121108-ns_1%40172.23.97.74.zip

      Attaching backup logs

      Attachments

        1. backup-0.log
          272 kB
          Arunkumar Senthilnathan

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-49331 [CBM] Authentication error when backing up during a rebalance

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-49332 [CBM] EOF when backing up during a rebalance

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-49334 [CBM] 'connection refused' when backing up during a rebalance

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-49374 [System Test][Backup Service] backup tasks failed with error - failed to process snapshot marker: received index create for vBucket 922 before opening

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-49375 [System Test][Backup Service] backup tasks failed with error - timeout reached whilst waiting for manifest id 123 to have propagated throughout the cluster, waited 1m0s

          • Major - Major loss of function.
          • Closed

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty