Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46561

[System Test][Backup Service] error observed when a local disk backup repo is used as staging directory for a cloud repo - Backup repository is corrupted 'rc2' directory is not in the correct format

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      7.0.0-5219

      Test:
      -test tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml
      Scale 3 Iteration 2

      Steps:
      create backup plan using /data/archive as the archive location (localdisk backup - every 6 hours - merge every 2 days)
      let couple of backups and merges go thru
      create another backup using a s3 bucket as the archive location but give /data/archive as the staging directory location (same as the archive location for localdisk backup)
      Trigger a backup on the cloud archive

      the cloud backup went thru but my localdisk repo became inaccessible - started seeing errors like this (both UI and logs):
      “

      {\“error\“:\“Backup repository is corrupted ‘rc2’ directory is not in the correct format\“}

      (rc2 is the folder name in cloud archive)

      .123.28:

      021-05-25T11:41:23.628-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:41:23.628-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:41:33.624-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:41:33.624-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:41:37.146-07:00 INFO (Stats) Start repositories data size collection
      		 
      2021-05-25T11:41:37.342-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info -j -a /data/archive --offset 0 --limit 0 -r ed601e0c-fafe-45d1-87df-553bf9936783
      		 
      2021-05-25T11:41:37.368-07:00 WARN (Stats) Failed to run info on repository {"repository": "my_repo", "err": "exit status 1", "out": "{\"error\":\"Backup repository is corrupted 'rc2' directory is not in the correct format\"}\n"}
      		 
      2021-05-25T11:41:37.368-07:00 INFO (Stats) Stop repositories data size collection
      		 
      2021-05-25T11:41:43.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:41:43.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:41:53.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:41:53.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:42:03.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:42:03.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:42:13.625-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:42:13.625-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:42:23.628-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:42:23.628-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:42:33.624-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:42:33.624-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:42:43.622-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:42:43.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:42:53.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:42:53.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:43:03.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:43:03.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:43:13.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:43:13.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:43:23.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:43:23.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:43:33.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:43:33.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:43:43.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:43:43.624-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:43:53.627-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:43:53.628-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:44:03.635-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:44:03.635-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:44:13.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:44:13.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:44:23.624-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:44:23.624-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:44:33.627-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:44:33.627-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:44:43.623-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:44:43.623-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:44:53.625-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:44:53.625-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:45:03.624-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:45:03.624-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "running"}
      		 
      2021-05-25T11:45:13.618-07:00 WARN (Worker) No progress given by cbbackupmgr {"cluster": "self", "repositoryID": "my_repo_s3", "state": "active", "taskName": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8"}
      		 
      2021-05-25T11:45:23.012-07:00 INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "my_repo_s3", "state": "active", "taskName": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8"}
      		 
      2021-05-25T11:45:23.014-07:00 INFO (Worker) Task done {"cluster": "self", "repositoryID": "my_repo_s3", "state": "active", "taskName": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8"}
      		 
      2021-05-25T11:45:23.018-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:45:23.018-07:00 DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "done"}
      		 
      2021-05-25T11:45:23.018-07:00 INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "my_repo_s3", "task": "BACKUP-a2cbb91c-0352-4d51-b1db-5962f6cb78e8", "status": "done"}
      		 
      2021-05-25T11:45:24.466-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
      		 
      2021-05-25T11:45:24.466-07:00 DEBUG (Leader Manager) Received store event {"eventType": 5}
      		 
      2021-05-25T11:45:24.467-07:00 DEBUG (Leader Manager) Received store event {"eventType": 3}
      		 
      2021-05-25T11:46:37.146-07:00 INFO (Stats) Start repositories data size collection
      		 
      2021-05-25T11:46:37.333-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info -j -a /data/archive --offset 0 --limit 0 -r ed601e0c-fafe-45d1-87df-553bf9936783
      		 
      2021-05-25T11:46:37.367-07:00 WARN (Stats) Failed to run info on repository {"repository": "my_repo", "err": "exit status 1", "out": "{\"error\":\"Backup repository is corrupted 'rc2' directory is not in the correct format\"}\n"}
      		 
      2021-05-25T11:46:37.367-07:00 INFO (Stats) Stop repositories data size collection

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.104.137.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.104.155.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.104.157.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.104.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.104.69.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.104.70.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.105.107.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.106.100.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.106.188.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.108.103.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.120.245.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.121.117.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.121.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.123.27.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.123.28.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.96.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.96.251.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.96.252.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.96.253.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.97.119.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.97.121.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.97.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.97.239.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.97.242.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.98.135.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.99.11.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.99.20.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.99.21.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621976855/collectinfo-2021-05-25T210737-ns_1%40172.23.99.25.zip

      Attaching backup log

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            arunkumar Arunkumar Senthilnathan (Inactive)
            arunkumar Arunkumar Senthilnathan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty