Errors in backup pod logs on successful exit of the script

Test: TestFullIncremental

SHA: 41d3fa4689c97409f80ebfabaf24a3ac7c268da4 (12th patch set from http://review.couchbase.org/#/c/122414)

Test result:

– PASS: TestOperator (231.54s)
=== RUN TestOperator/TestFullIncremental
— PASS: TestOperator/TestFullIncremental (185.83s)
crd_util.go:26: creating couchbase cluster: test-couchbase-2xztt
test_util.go:35: Suite Test Results:
test_util.go:64: 1: TestFullIncremental...PASS
test_util.go:106:
Pass: 1.000000
Fail: 0.000000
Pass Rate: 100.000000

Following errors observed in full-incremental-full pod logs even on successful exit of backup script:

2020-03-04 19:50:35,774 - root - INFO - removed 0 backups
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict(

{'Audit-Id': '5575ad8e-00b5-4a96-9bcb-d6a84d8519ca', 'Content-Type': 'application/json', 'Date': 'Wed, 04 Mar 2020 19:50:28 GMT', 'Content-Length': '246'}

)
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"couchbasebackups.couchbase.com \"cbbackup\" not found","reason":"NotFound","details":

{"name":"cbbackup","group":"couchbase.com","kind":"couchbasebackups"}

,"code":404}

(409)
Reason: Conflict
HTTP response headers: HTTPHeaderDict(

{'Audit-Id': 'c8b39fed-d57c-4f6b-8885-7b711aaa7ccc', 'Content-Type': 'application/json', 'Date': 'Wed, 04 Mar 2020 19:50:28 GMT', 'Content-Length': '378'}

)
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Operation cannot be fulfilled on couchbasebackups.couchbase.com \"full-incremental\": the object has been modified; please apply your changes to the latest version and try again","reason":"Conflict","details":

{"name":"full-incremental","group":"couchbase.com","kind":"couchbasebackups"}

,"code":409}

(422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict(

{'Audit-Id': 'b700dca7-ef09-455d-aa40-2ca07a998fb0', 'Content-Type': 'application/json', 'Date': 'Wed, 04 Mar 2020 19:50:28 GMT', 'Transfer-Encoding': 'chunked'}

)
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"CouchbaseBackup.couchbase.com \"full-incremental\" is invalid: []: Invalid value: map[string]interface {}{\"kind\":\"CouchbaseBackup\", \"metadata\":map[string]interface {}

{\"name\":\"full-incremental\", \"namespace\":\"default\", \"selfLink\":\"/apis/couchbase.com/v2/namespaces/default/couchbasebackups/full-incremental\", \"uid\":\"3c0df884-5e51-11ea-8ca9-02d96665db1c\", \"resourceVersion\":\"56079248\", \"generation\":11, \"creationTimestamp\":\"2020-03-04T19:49:18Z\"}

, \"spec\":map[string]interface {}{\"backoffLimit\":2, \"failedJobsHistoryLimit\":5, \"logRetention\":\"168h\", \"size\":\"2Gi\", \"full\":map[string]interface {}

{\"schedule\":\"*/5 * * * *\"}

, \"backupRetention\":\"720h\", \"incremental\":map[string]interface {}

{\"schedule\":\"*/2 * * * *\"}

, \"successfulJobsHistoryLimit\":3, \"strategy\":\"full_incremental\"}, \"status\":map[string]interface {}{\"running\":true, \"failed\":false, \"pod\":\"full-incremental-incremental-1583351400-6qf7x\", \"archive\":\"/data/backups\", \"capacityUsed\":\"0.01Gi\", \"cronjob\":\"full-incremental-incremental\", \"repoList\":[]interface {}

{\"test-couchbase-2xztt-2020-03-04T19_50_27\", \"test-couchbase-2xztt-2020-03-04T19_50_28\"}, \"job\":\"full-incremental-incremental-1583351400\", \"lastRun\":\"2020-03-04T19:50:27Z\", \"repo\":\"test-couchbase-2xztt-2020-03-04T19_50_28\"}, \"apiVersion\":\"couchbase.com/v2\"}: validation failure list:\nstatus.repoList in body must be of type list: \"array\"","reason":"Invalid","details":{"name":"full-incremental","group":"couchbase.com","kind":"CouchbaseBackup","causes":[{"reason":"FieldValueInvalid","message":"Invalid value: map[string]interface {}{\"status\":map[string]interface {}{\"running\":true, \"failed\":false, \"pod\":\"full-incremental-incremental-1583351400-6qf7x\", \"archive\":\"/data/backups\", \"capacityUsed\":\"0.01Gi\", \"cronjob\":\"full-incremental-incremental\", \"repoList\":[]interface {}{"test-couchbase-2xztt-2020-03-04T19_50_27", "test-couchbase-2xztt-2020-03-04T19_50_28"}

, \"job\":\"full-incremental-incremental-1583351400\", \"lastRun\":\"2020-03-04T19:50:27Z\", \"repo\":\"test-couchbase-2xztt-2020-03-04T19_50_28\"}, \"apiVersion\":\"couchbase.com/v2\", \"kind\":\"CouchbaseBackup\", \"metadata\":map[string]interface {}

{\"generation\":11, \"creationTimestamp\":\"2020-03-04T19:49:18Z\", \"name\":\"full-incremental\", \"namespace\":\"default\", \"selfLink\":\"/apis/couchbase.com/v2/namespaces/default/couchbasebackups/full-incremental\", \"uid\":\"3c0df884-5e51-11ea-8ca9-02d96665db1c\", \"resourceVersion\":\"56079248\"}

, \"spec\":map[string]interface {}{\"successfulJobsHistoryLimit\":3, \"strategy\":\"full_incremental\", \"backupRetention\":\"720h\", \"incremental\":map[string]interface {}

{\"schedule\":\"*/2 * * * *\"}

, \"logRetention\":\"168h\", \"size\":\"2Gi\", \"full\":map[string]interface {}

{\"schedule\":\"*/5 * * * *\"}

, \"backoffLimit\":2, \"failedJobsHistoryLimit\":5}}: validation failure list:\nstatus.repoList in body must be of type list: \"array\"","field":"[]"}]},"code":422}