Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48927

[System Test][System Event Logs]Backup service - error "timestamp":"The value must be a valid ISO 8601 UTC" observed

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • No

    Description

      7.1.0-1480

      We create a backup plan right at the beginning of our longevity test - observing the following errors in backup service logs:

      2021-10-14T04:02:23.164-07:00 INFO (HTTP Manager) Added plan {"name": "my_plan"}
      2021-10-14T04:02:23.164-07:00 DEBUG (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.120.74:8091/_event'
      2021-10-14T04:02:23.165-07:00 DEBUG (REST) (Attempt 1) (POST) (400) Received response from 'http://172.23.120.74:8091/_event'
      2021-10-14T04:02:23.165-07:00 WARN (REST) (Attempt 1) (POST) Request to endpoint '/_event' failed with status code 400
      2021-10-14T04:02:23.165-07:00 ERROR (elutil) Failed to report event 'elutil.Event{Component:"backup", Severity:"info", EventID:0x1801, Description:"A new plan was created", ExtraAttributes:map[string]interface {}{"name":"my_plan"}, SubComponent:""}' due to error 'unexpected status code 400 for 'POST' request to '/_event', {"errors":{"timestamp":"The value must be a valid ISO 8601 UTC"}}'
      2021-10-14T04:02:28.790-07:00 INFO (REST) POST /api/v1/cluster/self/repository/active/my_repo
      2021-10-14T04:02:28.804-07:00 INFO (GRPC Manager) Global storage location check {"req": "location:\"/data/archive/.cbbs-6c781164db5b9e35bb697e3f0a9a8e03/node-b28ca7e04496c3e8b074d9e4149139d7-1634209348798784420\"  sourceNode:\"b28ca7e04496c3e8b074d9e4149139d7\""}
      2021-10-14T04:02:28.809-07:00 INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr config -a /data/archive -r 3ded8fe6-6a52-4f0d-baf9-ced55cecff5b --cbbs --readme-author Backup Service, cluster: 6c781164db5b9e35bb697e3f0a9a8e03 --disable-views --disable-ft-indexes --disable-ft-alias --disable-eventing --disable-analytics --disable-cluster-analytics
      2021-10-14T04:02:28.853-07:00 DEBUG (Leader Manager) Received store event {"eventType": 4}
      2021-10-14T04:02:28.853-07:00 INFO (Event Handler) (Repo Change) Repository change {"event": {"Type":4,"Path":"/cbbs/cluster/self/active/my_repo","Value":"eyJpZCI6Im15X3JlcG8iLCJwbGFuX25hbWUiOiJteV9wbGFuIiwic3RhdGUiOiJhY3RpdmUiLCJhcmNoaXZlIjoiL2RhdGEvYXJjaGl2ZSIsInJlcG8iOiIzZGVkOGZlNi02YTUyLTRmMGQtYmFmOS1jZWQ1NWNlY2ZmNWIiLCJ2ZXJzaW9uIjoxLCJjcmVhdGlvbl90aW1lIjoiMjAyMS0xMC0xNFQwNDowMjoyOC44NDI4NzAxNzYtMDc6MDAiLCJ1cGRhdGVfdGltZSI6IjIwMjEtMTAtMTRUMDQ6MDI6MjguODQyODcwMTc2LTA3OjAwIn0=","Rev":"g2gCbgUAeobc2g5sAAAAAWgCbQAAACBiMjhjYTdlMDQ0OTZjM2U4YjA3NGQ5ZTQxNDkxMzlkN2gCYQFuBQBEitzaDmo="}}
      2021-10-14T04:02:28.854-07:00 INFO (Event Handler) (Repo Change) Resuming repository {"repositoryID": "my_repo", "planName": "my_plan"}
      2021-10-14T04:02:28.857-07:00 INFO (Manger) Added repository {"id": "my_repo", "cluster": "self", "archive": "/data/archive", "repo": "3ded8fe6-6a52-4f0d-baf9-ced55cecff5b", "plan": "my_plan"}
      2021-10-14T04:02:28.857-07:00 DEBUG (REST) (Attempt 1) (POST) Dispatching request to 'http://172.23.120.74:8091/_event'
      2021-10-14T04:02:28.858-07:00 INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "my_repo", "task": "backup-1"}
      2021-10-14T04:02:28.859-07:00 DEBUG (REST) (Attempt 1) (POST) (400) Received response from 'http://172.23.120.74:8091/_event'
      2021-10-14T04:02:28.859-07:00 WARN (REST) (Attempt 1) (POST) Request to endpoint '/_event' failed with status code 400
      2021-10-14T04:02:28.859-07:00 ERROR (elutil) Failed to report event 'elutil.Event{Component:"backup", Severity:"info", EventID:0x1807, Description:"A new repository has been created", ExtraAttributes:map[string]interface {}{"cluster":"self", "name":"my_repo", "plan":"my_plan"}, SubComponent:""}' due to error 'unexpected status code 400 for 'POST' request to '/_event', {"errors":{"timestamp":"The value must be a valid ISO 8601 UTC"}}'
      2021-10-14T04:02:28.899-07:00 INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "my_repo", "task": "merge"}
      

      Cluster config:
      Cluster config:
      kv : 12 ===== > [172.23.120.73:8091 172.23.120.77:8091 172.23.120.86:8091 172.23.121.77:8091 172.23.123.24:8091 172.23.123.25:8091 172.23.123.26:8091 172.23.96.122:8091 172.23.96.14:8091 172.23.96.48:8091 172.23.97.241:8091 172.23.97.74:8091]
      backup : 1 ===== > [172.23.120.74:8091]
      n1ql : 2 ===== > [172.23.96.254:8091 172.23.97.112:8091]
      index : 6 ===== > [172.23.120.58:8091 172.23.120.75:8091 172.23.120.81:8091 172.23.123.31:8091 172.23.123.32:8091 172.23.123.33:8091]

      Logs:
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.58.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.73.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.74.zip (backup node)
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.75.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.81.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.120.86.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.121.77.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.123.24.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.123.25.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.123.26.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.123.31.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.123.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.123.33.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.96.122.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.96.14.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.96.254.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.96.48.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.97.112.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.97.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1634209889/collectinfo-2021-10-14T111131-ns_1%40172.23.97.74.zip

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-48927
          # Subject Branch Project Status CR V

          Activity

            People

              arunkumar Arunkumar Senthilnathan (Inactive)
              arunkumar Arunkumar Senthilnathan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty