Patrick Varley added a comment - 29/Mar/21 2:55 PM The error in the backup log is 2021-03-29T11:57:58.029-07:00 WARN: (Couchbase) Unexpected error 'operation timed out after 5s' while trying to get sequence numbers, will retry in 5s -- couchbase.GetSequenceNumbers() at sequence_numbers.go:40 2021-03-29T11:58:09.070-07:00 (Plan) (Data) Successfully decided which key value data to transfer for bucket 'bucket-1' | {"number":11,"duration":"16.106882723s"} 2021-03-29T11:58:09.070-07:00 (Plan) (Data) Transferring new key value data for bucket 'bucket-1' 2021-03-29T11:58:09.070-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.105.72:8091/pools/default/buckets/bucket-1' 2021-03-29T11:58:09.079-07:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.105.72:8091/pools/default/buckets/bucket-1' 2021-03-29T11:58:09.079-07:00 (REST) (Attempt 1) (GET) Dispatching request to 'http://172.23.105.72:8091/pools/default/buckets/bucket-1/scopes' 2021-03-29T11:58:09.083-07:00 (REST) (Attempt 1) (GET) (200) Received response from 'http://172.23.105.72:8091/pools/default/buckets/bucket-1/scopes' 2021-03-29T11:58:09.084-07:00 (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.105.72:8091/pools/default/buckets/bucket-1/scopes/%40ensureManifest/0' 2021-03-29T11:58:09.090-07:00 (REST) (Attempt 1) (POST) (200) Received response from 'http://172.23.105.72:8091/pools/default/buckets/bucket-1/scopes/%40ensureManifest/0' 2021-03-29T11:59:10.716-07:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operation for bucket 'bucket-1': failed to transfer bucket data for bucket 'bucket-1': failed to transfer key value data: failed to transfer key value data: failed to initialise worker 11: failed to get gocbcore DCP agent: agent failed to connect to the cluster: unambiguous timeout | {"InnerError":{"InnerError":{"InnerError":{},"Message":"unambiguous timeout"}},"OperationID":"WaitUntilReady","Opaque":"","TimeObserved":60000110502,"RetryReasons":["NOT_READY"],"RetryAttempts":65,"LastDispatchedTo":"","LastDispatchedFrom":"","LastConnectionID":""} This suggest that cbbackupmgr got a timeout when trying to fetch the seqno from Couchbase Server. I believe the seqno number come directly form the underlying storage engine, so this might be a magma issue. The memcached.log will log slow operations, so it be worth check them, will need server logs for that case too.

Patrick Varley added a comment - 29/Mar/21 3:00 PM Bo-Chun Wang has there been failures reported on couchstore, I wonder if both issue this and MB-45320 are related to magma? Bo-Chun Wang reply No, the second defect (this one MB-45322 ) doesn't happen on magma only. Several performance runs on couchstore failed, too. The runs failed right after backup command was issued. I don't have logs for these runs. http://perf.jenkins.couchbase.com/job/leto/17806/ http://perf.jenkins.couchbase.com/job/leto/17809/ http://perf.jenkins.couchbase.com/job/leto/17815/ 2021-03-29T02:22:17 [INFO] Running: ./opt/couchbase/bin/cbbackupmgr backup --archive /data/workspace/backup --repo default --host http://leto-srv-01.perf.couchbase.com --username Administrator --password password --threads 16 --storage forestdb 19:22:17 Fatal error: local() encountered an error (return code 1) while executing './opt/couchbase/bin/cbbackupmgr backup --archive /data/workspace/backup --repo default --host http://leto-srv-01.perf.couchbase.com --username Administrator --password password --threads 16 --storage forestdb' 19:22:17 Aborting.

Bo-Chun Wang added a comment - 29/Mar/21 3:15 PM Patrick Varley Sorry, I messed up. Those couchstore runs above are running on 6.6.2 build, and they are using forestdb. They have nothing to do with this issue. Sorry that I made a mistake.

Bo-Chun Wang added a comment - 29/Mar/21 3:36 PM I re-ran the same test on couchstore, and it finished successfully. The couchstore run didn't hit  MB-45320 and  MB-45322 .

Daniel Owen added a comment - 29/Mar/21 10:57 PM Hi Bo-Chun Wang , Is this a new test, or did it recently start to fail? Changing component to Couchbase-bucket / storage-engine as issue appears to be with magma. Given that it appears to be a magma only issue - moving to CC.next

Bo-Chun Wang added a comment - 30/Mar/21 3:53 PM Daniel Owen No, it's not a new test. It was working fine until the latest weekly build (4797). Patrick commented in MB-45320 that the issue may be caused by  GOCBC-1073 . I have kicked off a run with build 4814 having the fix for  GOCBC-1073 and will update when I have results. 

Bo-Chun Wang added a comment - 30/Mar/21 9:10 PM I have a good run with build 4814. http://perf.jenkins.couchbase.com/job/rhea-5node1/896/  

Jyotsna Nayak added a comment - 30/Mar/21 9:13 PM HI ,  I am seeing a similar issue with backup for the FTS tests. Some of the tests are as follows :  1. http://perf.jenkins.couchbase.com/view/FTS/job/atlas_collections/461/console 2. http://perf.jenkins.couchbase.com/view/FTS/job/atlas_collections/461/console First build I observed this : 7.0.0-4726  for this build the backup was failing when run locally also. For build 7.7.0-4797 I am able to run the test locally with no failure with rebalance. But I am not able to run the same test via Jenkins

Jyotsna Nayak added a comment - 30/Mar/21 9:14 PM I have a good run also with build 7.7.0: 4814 link to the job: http://perf.jenkins.couchbase.com/view/FTS/job/atlas_collections/476/consoleFull    

Daniel Owen added a comment - 31/Mar/21 6:03 AM OK thanks Bo-Chun Wang , Jyotsna Nayak - so looks like it was a duplicate of GOCBC-1073 .

Mihir Kamdar added a comment - 01/Apr/21 4:20 PM Bulk closing non-fixed defects. Feel free to reopen if necessary.