2024-01-03T07:43:54.434Z INFO (Main) Running node version backup-7.6.0-1975-430de15 with options: -http-port=8097 -grpc-port=9124 -https-port=18097 -cert-path=/opt/couchbase/var/lib/couchbase/config/certs/chain.pem -key-path=/opt/couchbase/var/lib/couchbase/config/certs/pkey.pem -ca-path=/opt/couchbase/var/lib/couchbase/config/certs/ca.pem -client-cert-path=/opt/couchbase/var/lib/couchbase/config/certs/client_chain.pem -client-key-path=/opt/couchbase/var/lib/couchbase/config/certs/client_pkey.pem -ipv4=required -ipv6=optional -cbm=/opt/couchbase/bin/cbbackupmgr -node-uuid=80802b2860060fffd26a01149967b587 -public-address=127.0.0.1 -admin-port=8091 -log-file=none -log-level=debug -integrated-mode -integrated-mode-host=http://127.0.0.1:8091 -secure-integrated-mode-host=https://127.0.0.1:18091 -integrated-mode-user=@backup -tmp-dir=/opt/couchbase/var/lib/couchbase/tmp -cbauth-host=127.0.0.1:8091 2024-01-03T07:43:54.434Z INFO (Main) Initialized logger {"log level": "debug"} 2024-01-03T07:43:54.435Z DEBUG (Main) File limit info {"curr": 200000, "max": 200000, "err": null} 2024-01-03T07:43:54.436Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T07:43:54.438Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T07:43:54.438Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T07:43:54.440Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T07:43:54.440Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T07:43:54.440Z INFO (Node) Confirmed cluster uuid {"clusterUUID": "0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T07:43:54.598Z INFO (CBAUDIT) audit: created new audit service 2024-01-03T07:43:54.598Z INFO (Audit) Audit service initialised 2024-01-03T07:43:54.599Z INFO (Node) Starting GRPC manager 2024-01-03T07:43:54.601Z INFO (GRPC Manager) Starting GRPC Server {"address": "0.0.0.0:9124"} 2024-01-03T07:43:54.602Z INFO (GRPC Manager) Starting GRPC Server {"address": "[::]:9124"} 2024-01-03T07:43:54.609Z INFO (Node) Starting leader manager 2024-01-03T07:43:54.610Z INFO (Leader Manager) Starting leader manager 2024-01-03T07:43:54.610Z INFO (HTTP Manager) HTTP server started on {"address": "0.0.0.0:8097", "https": false} 2024-01-03T07:43:54.611Z INFO (HTTP Manager) HTTP server started on {"address": "[::]:8097", "https": false} 2024-01-03T07:43:54.612Z INFO (HTTP Manager) HTTP server started on {"address": "0.0.0.0:18097", "https": true} 2024-01-03T07:43:54.612Z INFO (HTTP Manager) HTTP server started on {"address": "[::]:18097", "https": true} 2024-01-03T07:43:54.614Z INFO (Storage) Starting MetaKV observer 2024-01-03T07:43:54.614Z INFO (Node) TLS config refreshed 2024-01-03T07:43:54.615Z WARN (Worker) Stopping task runner with 0 task on queue 2024-01-03T07:43:54.615Z WARN (Worker) Stopped 2024-01-03T07:43:54.615Z INFO (HTTP Manager) Shutting down http server {"https": true} 2024-01-03T07:43:54.615Z INFO (HTTP Manager) Shutting down http server {"https": false} 2024-01-03T07:43:54.615Z DEBUG (HTTP Manager) HTTP servers shutdown {"time_taken": "176.13µs"} 2024-01-03T07:43:54.618Z INFO (GRPC Manager) Starting GRPC Server {"address": "[::]:9124"} 2024-01-03T07:43:54.618Z INFO (GRPC Manager) Starting GRPC Server {"address": "0.0.0.0:9124"} 2024-01-03T07:43:54.623Z DEBUG (Leader Manager) Received store event {"eventType": 6} 2024-01-03T07:43:54.626Z INFO (HTTP Manager) HTTP server started on {"address": "[::]:8097", "https": false} 2024-01-03T07:43:54.626Z INFO (HTTP Manager) HTTP server started on {"address": "0.0.0.0:8097", "https": false} 2024-01-03T07:43:54.627Z INFO (HTTP Manager) HTTP server started on {"address": "0.0.0.0:18097", "https": true} 2024-01-03T07:43:54.627Z INFO (HTTP Manager) HTTP server started on {"address": "[::]:18097", "https": true} 2024-01-03T07:43:58.742Z INFO (Service Manager) Prepare topology change {"change": {"id":"e1e49cf7212cc82cf4f25c1e63b6e24f","currentTopologyRev":null,"type":"topology-change-rebalance","keepNodes":[{"nodeInfo":{"nodeId":"80802b2860060fffd26a01149967b587","priority":1,"opaque":{"grpc_port":9124,"host":"127.0.0.1","http_port":8097}},"recoveryType":"recovery-full"}],"ejectNodes":[]}} 2024-01-03T07:43:58.746Z INFO (Service Manager) Start topology change {"change": {"id":"e1e49cf7212cc82cf4f25c1e63b6e24f","currentTopologyRev":null,"type":"topology-change-rebalance","keepNodes":[{"nodeInfo":{"nodeId":"80802b2860060fffd26a01149967b587","priority":1,"opaque":{"grpc_port":9124,"host":"127.0.0.1","http_port":8097}},"recoveryType":"recovery-full"}],"ejectNodes":[]}} 2024-01-03T07:43:58.746Z INFO (Rebalance) Starting rebalance {"change": {"id":"e1e49cf7212cc82cf4f25c1e63b6e24f","currentTopologyRev":null,"type":"topology-change-rebalance","keepNodes":[{"nodeInfo":{"nodeId":"80802b2860060fffd26a01149967b587","priority":1,"opaque":{"grpc_port":9124,"host":"127.0.0.1","http_port":8097}},"recoveryType":"recovery-full"}],"ejectNodes":[]}} 2024-01-03T07:43:58.751Z INFO (Rebalance) Got old leader {"leader": ""} 2024-01-03T07:43:58.756Z INFO (Rebalance) Got current nodes {"#nodes": 0} 2024-01-03T07:43:58.756Z INFO (Rebalance) Setting self as leader 2024-01-03T07:43:58.758Z DEBUG (Leader Manager) Received store event {"eventType": 1} 2024-01-03T07:43:58.758Z INFO (Leader Manager) Stepped up as leader 2024-01-03T07:43:58.758Z INFO (Leader Manager) Leader value changed {"leader": true} 2024-01-03T07:43:58.758Z INFO (Janitor) Starting janitor 2024-01-03T07:43:58.758Z INFO (Stats) Starting periodic stat collection {"period": "5m0s"} 2024-01-03T07:43:58.758Z INFO (Leader Manager) Constructing schedule from storage 2024-01-03T07:43:58.758Z INFO (Janitor) Starting time jump checker 2024-01-03T07:43:58.759Z INFO (Rebalance) Did the failover nodes will do eject node now {"eject nodes": []} 2024-01-03T07:43:58.759Z INFO (Rebalance) Adding nodes now {"keep nodes": [{"nodeInfo":{"nodeId":"80802b2860060fffd26a01149967b587","priority":1,"opaque":{"grpc_port":9124,"host":"127.0.0.1","http_port":8097}},"recoveryType":"recovery-full"}]} 2024-01-03T07:43:58.759Z INFO (Rebalance) Keep node {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T07:43:58.759Z DEBUG (Rebalance) Added node to MetaKV {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T07:43:58.762Z DEBUG (Rebalance) Checking if status match {"expected": "node in"} 2024-01-03T07:43:58.769Z DEBUG (Leader Manager) Received store event {"eventType": 2} 2024-01-03T07:43:58.769Z INFO (Event Handler) (Topology) Node has been added to the cluster 2024-01-03T07:43:58.769Z INFO (Leader Manager) Switch status {"nodeStatus": "node in"} 2024-01-03T07:44:03.762Z DEBUG (Rebalance) Checking for rebalance status match failed {"attempt": 1, "error": "node status is not in", "expected": "node in"} 2024-01-03T07:44:03.762Z INFO (Rebalance) Nodes added 2024-01-03T07:44:03.762Z INFO (ClockKeeper) Starting scheduler 2024-01-03T07:44:03.762Z INFO (Rebalance) Rebalance done {"err": null, "state": {}, "cancelled": false} 2024-01-03T07:44:03.762Z INFO (Janitor) Storage access check triggered 2024-01-03T07:44:03.762Z INFO (Janitor) Running storage access check 2024-01-03T07:44:03.764Z INFO (Janitor) Running storage access check on state active 2024-01-03T07:44:03.765Z INFO (Janitor) Running storage access check on state archived 2024-01-03T07:46:48.965Z INFO (REST) POST /api/v1/cluster/self/repository/active/repo_jan3_07_45_utc 2024-01-03T07:46:48.968Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T07:46:48.969Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T07:46:48.969Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T07:46:48.970Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T07:46:48.970Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T07:46:48.970Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T07:46:48.974Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T07:46:49.014Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704268008978085830\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T07:46:49.016Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr config --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --cbbs --readme-author Backup Service, cluster: 0a00a7717f426e18e6cd4bd07e79f0e7 --include-data travel-sample 2024-01-03T07:46:49.054Z DEBUG (Leader Manager) Received store event {"eventType": 4} 2024-01-03T07:46:49.055Z INFO (Event Handler) (Repo Change) Repository change {"event": {"Type":4,"Path":"/cbbs/cluster/self/active/repo_jan3_07_45_utc","Value":"eyJpZCI6InJlcG9famFuM18wN180NV91dGMiLCJwbGFuX25hbWUiOiJfaG91cmx5X2JhY2t1cHMiLCJidWNrZXQiOnsibmFtZSI6InRyYXZlbC1zYW1wbGUiLCJ1dWlkIjoiODgwNjUxZDM5ODM4ZDkxYmY0MmMxMzRhOGQyMzU1OTcifSwic3RhdGUiOiJhY3RpdmUiLCJhcmNoaXZlIjoiL2hvbWUvZWMyLXVzZXIvY2JfYmFja3VwXzIwMjQwMTAxIiwicmVwbyI6IjAyMzExMGNmLTc4ZjktNDk0ZC1hMDIzLTk4MjM1NmNjNjI2MCIsInZlcnNpb24iOjEsImNyZWF0aW9uX3RpbWUiOiIyMDI0LTAxLTAzVDA3OjQ2OjQ5LjA1MDE2NzY1NVoiLCJ1cGRhdGVfdGltZSI6IjIwMjQtMDEtMDNUMDc6NDY6NDkuMDUwMTY3NjU1WiJ9","Rev":"g2gCbgUAhYsJ3w5sAAAAAWgCbQAAACA4MDgwMmIyODYwMDYwZmZmZDI2YTAxMTQ5OTY3YjU4N2gCYQFuBQDpjAnfDmo="}} 2024-01-03T07:46:49.055Z INFO (Event Handler) (Repo Change) Resuming repository {"repositoryID": "repo_jan3_07_45_utc", "planName": "_hourly_backups"} 2024-01-03T07:46:49.059Z INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T07:46:49.060Z INFO (Manger) Added repository {"id": "repo_jan3_07_45_utc", "cluster": "self", "archive": "/home/ec2-user/cb_backup_20240101", "repo": "023110cf-78f9-494d-a023-982356cc6260", "plan": "_hourly_backups"} 2024-01-03T07:46:49.073Z INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_every_6_hours"} 2024-01-03T07:46:49.085Z INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_week"} 2024-01-03T07:47:02.246Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T07:48:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T07:48:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T07:48:58.815Z INFO (Stats) Stop repositories data size collection 2024-01-03T07:53:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T07:53:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T07:53:58.817Z INFO (Stats) Stop repositories data size collection 2024-01-03T07:58:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T07:58:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T07:58:58.812Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:01:07.549Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:01:13.599Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 10 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:01:30.806Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:03:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T08:03:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:03:58.814Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:08:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T08:08:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:08:58.817Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T08:13:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:13:58.813Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:18:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T08:18:58.773Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:18:58.817Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:23:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T08:23:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:23:58.811Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:28:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T08:28:58.775Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:28:58.818Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:33:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T08:33:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:33:58.793Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:38:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T08:38:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:38:58.796Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:43:58.762Z INFO (Janitor) Running storage access check 2024-01-03T08:43:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T08:43:58.762Z INFO (Janitor) Running orphan task detection 2024-01-03T08:43:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:43:58.771Z INFO (Janitor) Running storage access check on state active 2024-01-03T08:43:58.791Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704271438773404430\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T08:43:58.796Z INFO (Janitor) Running storage access check on state archived 2024-01-03T08:43:58.813Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:48:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T08:48:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:48:58.789Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:53:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T08:53:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:53:58.792Z INFO (Stats) Stop repositories data size collection 2024-01-03T08:58:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T08:58:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T08:58:58.817Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:00:54.073Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T10:00:54Z"}} 2024-01-03T09:00:54.073Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T09:00:54.083Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T09:00:54.084Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T09:00:54.085Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T09:00:54.091Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T09:00:54.091Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T09:00:54.091Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T09:00:54.099Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T09:00:54.111Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T09:00:54.117Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T09:00:54.126Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T09:00:54.128Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T09:00:54.128Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T09:00:54.131Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T09:00:54.131Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T09:00:54.131Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T09:00:54.135Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T09:00:54.135Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T09:00:54.138Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T09:00:54.139Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T09:00:54.140Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T09:00:54.141Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T09:00:54.101470609Z","error_code":0,"type":"BACKUP"}} 2024-01-03T09:00:54.147Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T09:00:54.147Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T09:00:54.148Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 279a6e74-74d7-4d4e-b0e7-e712fc9d6361 2024-01-03T09:00:54.155Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T09:00:54.182Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T09:00:54.184Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T09:01:04.159Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T09:01:04.159Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T09:01:12.773Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T09:01:12.773Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T09:01:12.776Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T09:01:12.776Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T09:01:12.776Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T09:01:12.808Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T09:01:12.808Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T09:01:12.808Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T09:03:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T09:03:58.775Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:03:58.902Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:08:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T09:08:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:08:58.883Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T09:13:58.772Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:13:58.870Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:18:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T09:18:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:18:58.868Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:23:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T09:23:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:23:58.875Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:28:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T09:28:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:28:58.894Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:33:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T09:33:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:33:58.912Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:38:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T09:38:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:38:58.857Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:43:58.761Z INFO (Janitor) Running orphan task detection 2024-01-03T09:43:58.761Z INFO (Janitor) Running storage access check 2024-01-03T09:43:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T09:43:58.785Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:43:58.786Z INFO (Janitor) Running storage access check on state active 2024-01-03T09:43:58.801Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704275038789281377\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T09:43:58.804Z INFO (Janitor) Running storage access check on state archived 2024-01-03T09:43:58.884Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:48:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T09:48:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:48:58.864Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:53:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T09:53:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:53:58.838Z INFO (Stats) Stop repositories data size collection 2024-01-03T09:58:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T09:58:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T09:58:58.880Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:00:54.330Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T11:00:54Z"}} 2024-01-03T10:00:54.330Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T10:00:54.338Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T10:00:54.340Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T10:00:54.341Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T10:00:54.342Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T10:00:54.342Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T10:00:54.342Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T10:00:54.346Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T10:00:54.358Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T10:00:54.363Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T10:00:54.369Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T10:00:54.370Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T10:00:54.370Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T10:00:54.372Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T10:00:54.373Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T10:00:54.373Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T10:00:54.376Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T10:00:54.377Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T10:00:54.378Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T10:00:54.378Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T10:00:54.379Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T10:00:54.380Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T10:00:54.348347773Z","error_code":0,"type":"BACKUP"}} 2024-01-03T10:00:54.380Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 4de26441-3536-48d9-b4f1-5dc10b1bd187 2024-01-03T10:00:54.382Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T10:00:54.383Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T10:00:54.386Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T10:00:54.429Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T10:00:54.432Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T10:01:02.819Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T10:01:02.819Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T10:01:02.823Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T10:01:02.823Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T10:01:02.824Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T10:01:02.843Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T10:01:02.844Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T10:01:02.844Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T10:03:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:03:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:03:58.900Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:08:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T10:08:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:08:58.859Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:13:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:13:58.854Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:18:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:18:58.777Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:18:58.879Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:23:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T10:23:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:23:58.878Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:28:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T10:28:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:28:58.867Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:33:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T10:33:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:33:58.861Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:38:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:38:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:38:58.845Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:43:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:43:58.761Z INFO (Janitor) Running storage access check 2024-01-03T10:43:58.761Z INFO (Janitor) Running orphan task detection 2024-01-03T10:43:58.773Z INFO (Janitor) Running storage access check on state active 2024-01-03T10:43:58.774Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:43:58.781Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704278638776514156\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T10:43:58.784Z INFO (Janitor) Running storage access check on state archived 2024-01-03T10:43:58.874Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:48:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:48:58.775Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:48:58.904Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:53:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:53:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:53:58.864Z INFO (Stats) Stop repositories data size collection 2024-01-03T10:58:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T10:58:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T10:58:58.885Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:00:54.576Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T12:00:54Z"}} 2024-01-03T11:00:54.576Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T11:00:54.583Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T11:00:54.584Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T11:00:54.584Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T11:00:54.585Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T11:00:54.585Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T11:00:54.585Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T11:00:54.588Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T11:00:54.595Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T11:00:54.599Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T11:00:54.604Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T11:00:54.604Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T11:00:54.604Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T11:00:54.606Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T11:00:54.606Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T11:00:54.606Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T11:00:54.608Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T11:00:54.609Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T11:00:54.610Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T11:00:54.610Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T11:00:54.610Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T11:00:54.611Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T11:00:54.589687401Z","error_code":0,"type":"BACKUP"}} 2024-01-03T11:00:54.614Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T11:00:54.614Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T11:00:54.616Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T11:00:54.616Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T11:00:54.617Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 72082c97-4d0b-4913-8da0-4cb5572f5a71 2024-01-03T11:00:54.628Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T11:01:02.748Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T11:01:02.748Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T11:01:02.751Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T11:01:02.751Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T11:01:02.751Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T11:01:02.764Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T11:01:02.764Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T11:01:02.764Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T11:03:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:03:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:03:58.855Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:08:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T11:08:58.773Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:08:58.914Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:13:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T11:13:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:13:58.906Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:18:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:18:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:18:58.900Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:23:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:23:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:23:58.860Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:28:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:28:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:28:58.867Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:33:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:33:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:33:58.905Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:38:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T11:38:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:38:58.854Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:43:58.761Z INFO (Janitor) Running orphan task detection 2024-01-03T11:43:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:43:58.761Z INFO (Janitor) Running storage access check 2024-01-03T11:43:58.773Z INFO (Janitor) Running storage access check on state active 2024-01-03T11:43:58.776Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:43:58.781Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704282238774891205\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T11:43:58.783Z INFO (Janitor) Running storage access check on state archived 2024-01-03T11:43:58.865Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:48:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T11:48:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:48:58.893Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:53:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T11:53:58.773Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:53:58.918Z INFO (Stats) Stop repositories data size collection 2024-01-03T11:58:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T11:58:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T11:58:58.911Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:00:54.793Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T13:00:54Z"}} 2024-01-03T12:00:54.793Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T12:00:54.799Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T12:00:54.800Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T12:00:54.800Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T12:00:54.801Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T12:00:54.801Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T12:00:54.801Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T12:00:54.804Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T12:00:54.811Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T12:00:54.816Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T12:00:54.820Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T12:00:54.821Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T12:00:54.822Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T12:00:54.823Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T12:00:54.823Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T12:00:54.823Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T12:00:54.826Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T12:00:54.827Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T12:00:54.828Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T12:00:54.828Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T12:00:54.829Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T12:00:54.830Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T12:00:54.805955287Z","error_code":0,"type":"BACKUP"}} 2024-01-03T12:00:54.833Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T12:00:54.833Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T12:00:54.834Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 677dc819-627d-443a-9698-d20b77f1b1d4 2024-01-03T12:00:54.835Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T12:00:54.849Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T12:00:54.852Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T12:01:03.325Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T12:01:03.325Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T12:01:03.328Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T12:01:03.328Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T12:01:03.328Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T12:01:03.347Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T12:01:03.347Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T12:01:03.347Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T12:03:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T12:03:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:03:58.879Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:08:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T12:08:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:08:58.914Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T12:13:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:13:58.904Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:18:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T12:18:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:18:58.868Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:23:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T12:23:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:23:58.880Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:28:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T12:28:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:28:58.862Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:33:58.762Z INFO (Stats) Start repositories data size collection 2024-01-03T12:33:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:33:58.885Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:38:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T12:38:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:38:58.899Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:43:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T12:43:58.758Z INFO (Janitor) Running storage access check 2024-01-03T12:43:58.758Z INFO (Janitor) Running orphan task detection 2024-01-03T12:43:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:43:58.769Z INFO (Janitor) Running storage access check on state active 2024-01-03T12:43:58.776Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704285838771005512\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T12:43:58.778Z INFO (Janitor) Running storage access check on state archived 2024-01-03T12:43:58.876Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:48:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T12:48:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:48:58.870Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:53:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T12:53:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:53:58.871Z INFO (Stats) Stop repositories data size collection 2024-01-03T12:58:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T12:58:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T12:58:58.861Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:00:54.988Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T14:00:54Z"}} 2024-01-03T13:00:54.988Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T13:00:54.995Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T13:00:54.996Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T13:00:54.996Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T13:00:54.997Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T13:00:54.997Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T13:00:54.997Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T13:00:55.001Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T13:00:55.007Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T13:00:55.010Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T13:00:55.016Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T13:00:55.017Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T13:00:55.017Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T13:00:55.018Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T13:00:55.018Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T13:00:55.018Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T13:00:55.020Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T13:00:55.021Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T13:00:55.022Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T13:00:55.024Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T13:00:55.002193779Z","error_code":0,"type":"BACKUP"}} 2024-01-03T13:00:55.024Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress fd596779-a52f-4b4e-a59e-5152368551c6 2024-01-03T13:00:55.026Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T13:00:55.027Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T13:00:55.027Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T13:00:55.027Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T13:00:55.033Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T13:00:55.045Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T13:00:55.045Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T13:01:03.349Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T13:01:03.349Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T13:01:03.350Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T13:01:03.350Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T13:01:03.350Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T13:01:03.366Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T13:01:03.366Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T13:01:03.366Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T13:03:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:03:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:03:58.878Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:08:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T13:08:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:08:58.893Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:13:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:13:58.882Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:18:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:18:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:18:58.935Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:23:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T13:23:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:23:58.887Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:28:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T13:28:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:28:58.936Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:33:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:33:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:33:58.893Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:38:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T13:38:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:38:58.934Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:43:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:43:58.761Z INFO (Janitor) Running orphan task detection 2024-01-03T13:43:58.761Z INFO (Janitor) Running storage access check 2024-01-03T13:43:58.770Z INFO (Janitor) Running storage access check on state active 2024-01-03T13:43:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:43:58.780Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704289438773798708\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T13:43:58.783Z INFO (Janitor) Running storage access check on state archived 2024-01-03T13:43:58.922Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:48:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:48:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:48:58.930Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:53:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T13:53:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:53:58.898Z INFO (Stats) Stop repositories data size collection 2024-01-03T13:58:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T13:58:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T13:58:58.890Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:00:55.188Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T15:00:55Z"}} 2024-01-03T14:00:55.188Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T14:00:55.197Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:00:55.198Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:00:55.199Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T14:00:55.200Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T14:00:55.200Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T14:00:55.200Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:00:55.204Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:00:55.212Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T14:00:55.217Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T14:00:55.223Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:00:55.223Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:00:55.223Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T14:00:55.224Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T14:00:55.224Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T14:00:55.224Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:00:55.227Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:00:55.228Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T14:00:55.229Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:00:55.229Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T14:00:55.230Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T14:00:55.232Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T14:00:55.206167873Z","error_code":0,"type":"BACKUP"}} 2024-01-03T14:00:55.234Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 7017493c-44c1-4147-b29e-9ed55db99749 2024-01-03T14:00:55.235Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:00:55.235Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T14:00:55.238Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T14:00:55.258Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:00:55.259Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:01:03.588Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T14:01:03.588Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T14:01:03.590Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:01:03.591Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T14:01:03.591Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T14:01:03.611Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:01:03.611Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:01:03.611Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:03:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T14:03:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:03:58.892Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:08:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T14:08:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:08:58.892Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T14:13:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:13:58.894Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:18:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T14:18:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:18:58.902Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:23:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T14:23:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:23:58.903Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:28:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T14:28:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:28:58.895Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:30:55.281Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"merge_every_6_hours","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T20:30:55Z"}} 2024-01-03T14:30:55.281Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "merge_every_6_hours"} 2024-01-03T14:30:55.288Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:30:55.289Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:30:55.290Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T14:30:55.290Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T14:30:55.291Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T14:30:55.291Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:30:55.294Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:30:55.300Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T14:30:55.304Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"merge_every_6_hours\""} 2024-01-03T14:30:55.309Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:30:55.310Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T14:30:55.310Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T14:30:55.311Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T14:30:55.311Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T14:30:55.312Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:30:55.314Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T14:30:55.315Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_every_6_hours"} 2024-01-03T14:30:55.316Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:30:55.316Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_every_6_hours", "status": "waiting"} 2024-01-03T14:30:55.317Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "merge_every_6_hours", "taskType": "MERGE"} 2024-01-03T14:30:55.318Z INFO (Dispatcher) Set running task {"task": {"task_name":"merge_every_6_hours","status":"waiting","start":"2024-01-03T14:30:55.294964239Z","error_code":0,"type":"MERGE"}} 2024-01-03T14:30:55.318Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 --start 2024-01-02T00:00:00Z --end 2024-01-02T23:59:59.999999999Z 2024-01-03T14:30:55.319Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:30:55.319Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_every_6_hours", "status": "running"} 2024-01-03T14:30:55.320Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T14:30:55.331Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:30:55.331Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:30:55.364Z ERROR (Worker) Could not run task {"err": "failed to run info: exit status 1", "clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "taskName": "merge_every_6_hours"} 2024-01-03T14:30:55.366Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:30:55.367Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_every_6_hours", "status": "failed"} 2024-01-03T14:30:55.367Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "merge_every_6_hours", "status": "failed"} 2024-01-03T14:30:55.386Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:30:55.386Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T14:30:55.387Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T14:33:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T14:33:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:33:58.901Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:38:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T14:38:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:38:58.903Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:43:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T14:43:58.758Z INFO (Janitor) Running orphan task detection 2024-01-03T14:43:58.758Z INFO (Janitor) Running storage access check 2024-01-03T14:43:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:43:58.770Z INFO (Janitor) Running storage access check on state active 2024-01-03T14:43:58.780Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704293038774186988\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T14:43:58.783Z INFO (Janitor) Running storage access check on state archived 2024-01-03T14:43:58.904Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:48:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T14:48:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:48:58.881Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:53:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T14:53:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:53:58.907Z INFO (Stats) Stop repositories data size collection 2024-01-03T14:58:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T14:58:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T14:58:58.903Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:00:55.377Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T16:00:55Z"}} 2024-01-03T15:00:55.377Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T15:00:55.389Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T15:00:55.390Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T15:00:55.391Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T15:00:55.392Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T15:00:55.392Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T15:00:55.392Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T15:00:55.395Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T15:00:55.402Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T15:00:55.406Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T15:00:55.414Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T15:00:55.415Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T15:00:55.416Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T15:00:55.417Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T15:00:55.417Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T15:00:55.417Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T15:00:55.421Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T15:00:55.421Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T15:00:55.422Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T15:00:55.424Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T15:00:55.424Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T15:00:55.425Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T15:00:55.396301653Z","error_code":0,"type":"BACKUP"}} 2024-01-03T15:00:55.426Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress de49b023-679e-4b75-acee-3f00b2125ea5 2024-01-03T15:00:55.427Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T15:00:55.427Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T15:00:55.443Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T15:00:55.466Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T15:00:55.466Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T15:01:03.268Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T15:01:03.268Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T15:01:03.270Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T15:01:03.270Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T15:01:03.270Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T15:01:03.283Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T15:01:03.283Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T15:01:03.283Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T15:03:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T15:03:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:03:58.905Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:08:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T15:08:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:08:58.935Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:13:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T15:13:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:13:58.936Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:18:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T15:18:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:18:58.926Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:23:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T15:23:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:23:58.907Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:28:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T15:28:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:28:58.928Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:33:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T15:33:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:33:58.943Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:38:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T15:38:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:38:58.913Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:43:58.760Z INFO (Janitor) Running orphan task detection 2024-01-03T15:43:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T15:43:58.760Z INFO (Janitor) Running storage access check 2024-01-03T15:43:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:43:58.771Z INFO (Janitor) Running storage access check on state active 2024-01-03T15:43:58.784Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704296638777847812\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T15:43:58.787Z INFO (Janitor) Running storage access check on state archived 2024-01-03T15:43:58.948Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:48:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T15:48:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:48:58.918Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:53:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T15:53:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:53:58.915Z INFO (Stats) Stop repositories data size collection 2024-01-03T15:58:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T15:58:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T15:58:58.939Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:00:55.581Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T17:00:55Z"}} 2024-01-03T16:00:55.581Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T16:00:55.599Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T16:00:55.601Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T16:00:55.601Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T16:00:55.602Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T16:00:55.602Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T16:00:55.602Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T16:00:55.606Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T16:00:55.618Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T16:00:55.623Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T16:00:55.631Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T16:00:55.633Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T16:00:55.633Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T16:00:55.634Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T16:00:55.634Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T16:00:55.634Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T16:00:55.640Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T16:00:55.640Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T16:00:55.641Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T16:00:55.641Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T16:00:55.642Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T16:00:55.644Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T16:00:55.609828225Z","error_code":0,"type":"BACKUP"}} 2024-01-03T16:00:55.645Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress f429dd6b-d324-476f-88e0-aa60328e4106 2024-01-03T16:00:55.646Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T16:00:55.646Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T16:00:55.648Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T16:00:55.662Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T16:00:55.662Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T16:01:03.994Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T16:01:03.995Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T16:01:03.999Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T16:01:03.999Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T16:01:03.999Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T16:01:04.014Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T16:01:04.014Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T16:01:04.014Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T16:03:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T16:03:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:03:58.954Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:08:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T16:08:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:08:58.908Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:13:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T16:13:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:13:58.929Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:18:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T16:18:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:18:58.923Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:23:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T16:23:58.770Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:23:58.914Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:28:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T16:28:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:28:58.939Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:33:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T16:33:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:33:58.938Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:38:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T16:38:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:38:58.924Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:43:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T16:43:58.760Z INFO (Janitor) Running orphan task detection 2024-01-03T16:43:58.760Z INFO (Janitor) Running storage access check 2024-01-03T16:43:58.769Z INFO (Janitor) Running storage access check on state active 2024-01-03T16:43:58.772Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:43:58.776Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704300238771301992\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T16:43:58.778Z INFO (Janitor) Running storage access check on state archived 2024-01-03T16:43:58.934Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:48:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T16:48:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:48:58.905Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:53:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T16:53:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:53:58.923Z INFO (Stats) Stop repositories data size collection 2024-01-03T16:58:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T16:58:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T16:58:58.908Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:00:55.782Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T18:00:55Z"}} 2024-01-03T17:00:55.782Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T17:00:55.787Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T17:00:55.788Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T17:00:55.788Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T17:00:55.789Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T17:00:55.789Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T17:00:55.789Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T17:00:55.792Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T17:00:55.799Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T17:00:55.803Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T17:00:55.807Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T17:00:55.808Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T17:00:55.808Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T17:00:55.809Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T17:00:55.810Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T17:00:55.810Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T17:00:55.812Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T17:00:55.812Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T17:00:55.814Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T17:00:55.814Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T17:00:55.815Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T17:00:55.815Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T17:00:55.793924593Z","error_code":0,"type":"BACKUP"}} 2024-01-03T17:00:55.818Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress b12e2bba-615b-4d6a-b1da-a416cf6abc2a 2024-01-03T17:00:55.818Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T17:00:55.818Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T17:00:55.819Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T17:00:55.830Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T17:00:55.830Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T17:01:04.150Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T17:01:04.151Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T17:01:04.153Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T17:01:04.153Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T17:01:04.153Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T17:01:04.170Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T17:01:04.170Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T17:01:04.170Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T17:03:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:03:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:03:58.939Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:08:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T17:08:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:08:58.930Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:12:04.086Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:12:06.958Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 10 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:13:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:13:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:13:58.965Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:18:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:18:58.773Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:18:59.014Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:23:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:23:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:23:58.923Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:28:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:28:58.767Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:28:58.966Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:33:58.761Z INFO (Stats) Start repositories data size collection 2024-01-03T17:33:58.768Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:33:58.973Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:38:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:38:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:38:58.923Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:41:17.641Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 10 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:43:58.760Z INFO (Janitor) Running storage access check 2024-01-03T17:43:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:43:58.760Z INFO (Janitor) Running orphan task detection 2024-01-03T17:43:58.772Z INFO (Janitor) Running storage access check on state active 2024-01-03T17:43:58.773Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:43:58.780Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704303838774617645\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T17:43:58.782Z INFO (Janitor) Running storage access check on state archived 2024-01-03T17:43:58.944Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:48:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:48:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:48:59.009Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:53:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T17:53:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:53:58.954Z INFO (Stats) Stop repositories data size collection 2024-01-03T17:58:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T17:58:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T17:58:58.983Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:00:42.645Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 10 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:00:55.633Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:00:56.025Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T19:00:56Z"}} 2024-01-03T18:00:56.025Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T18:00:56.032Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T18:00:56.033Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T18:00:56.033Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T18:00:56.034Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T18:00:56.034Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T18:00:56.034Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T18:00:56.037Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T18:00:56.044Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T18:00:56.048Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T18:00:56.052Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T18:00:56.053Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T18:00:56.054Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T18:00:56.055Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T18:00:56.055Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T18:00:56.055Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T18:00:56.058Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T18:00:56.058Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T18:00:56.060Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T18:00:56.061Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T18:00:56.061Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T18:00:56.062Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T18:00:56.039212852Z","error_code":0,"type":"BACKUP"}} 2024-01-03T18:00:56.063Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T18:00:56.064Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T18:00:56.065Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 1c786477-df09-4cae-8aa6-06a62f3d8557 2024-01-03T18:00:56.070Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T18:00:56.080Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T18:00:56.081Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T18:01:04.615Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T18:01:04.615Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T18:01:04.618Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T18:01:04.618Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T18:01:04.618Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T18:01:04.630Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T18:01:04.630Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T18:01:04.630Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T18:03:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:03:58.762Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:03:58.992Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:08:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:08:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:08:58.963Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:13:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T18:13:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:13:58.962Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:18:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:18:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:18:58.949Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:23:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:23:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:23:58.924Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:28:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:28:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:28:58.942Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:33:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:33:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:33:58.956Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:38:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T18:38:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:38:58.969Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:43:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T18:43:58.760Z INFO (Janitor) Running storage access check 2024-01-03T18:43:58.760Z INFO (Janitor) Running orphan task detection 2024-01-03T18:43:58.769Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:43:58.772Z INFO (Janitor) Running storage access check on state active 2024-01-03T18:43:58.781Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704307438774087013\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T18:43:58.783Z INFO (Janitor) Running storage access check on state archived 2024-01-03T18:43:59.006Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:48:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:48:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:48:58.932Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:53:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T18:53:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:53:58.954Z INFO (Stats) Stop repositories data size collection 2024-01-03T18:58:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T18:58:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T18:58:58.940Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:00:56.267Z DEBUG (ClockKeeper) Job triggered {"job": {"TaskName":"backup_hourly","ClusterName":"self","RepositoryName":"repo_jan3_07_45_utc","NextScheduledRun":"2024-01-03T20:00:56Z"}} 2024-01-03T19:00:56.267Z INFO (Dispatcher) Dispatching task {"clusterName": "self", "repositoryID": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T19:00:56.277Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T19:00:56.278Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T19:00:56.278Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T19:00:56.280Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T19:00:56.280Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T19:00:56.280Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T19:00:56.283Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T19:00:56.291Z INFO (Dispatcher) Dispatching task {"targetNode": {"node_id":"80802b2860060fffd26a01149967b587","host":"127.0.0.1","http_port":8097,"grpc_port":9124,"status":"","leader":false}} 2024-01-03T19:00:56.295Z INFO (GRPC Manager) Received run order {"order": "cluster_name:\"self\" repository_id:\"repo_jan3_07_45_utc\" task_name:\"backup_hourly\""} 2024-01-03T19:00:56.301Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T19:00:56.302Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://127.0.0.1:8091/pools/default/nodeServices' 2024-01-03T19:00:56.302Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools' 2024-01-03T19:00:56.303Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools' 2024-01-03T19:00:56.304Z INFO (REST) Successfully connected to cluster | {"enterprise":true,"uuid":"0a00a7717f426e18e6cd4bd07e79f0e7"} 2024-01-03T19:00:56.304Z DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T19:00:56.306Z DEBUG (REST) (Attempt 1) (GET) (200) Received response from 'http://localhost:8091/pools/default/buckets/travel-sample' 2024-01-03T19:00:56.306Z INFO (Worker) Adding task to queue {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly"} 2024-01-03T19:00:56.308Z INFO (Worker) Running task {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly", "taskType": "BACKUP"} 2024-01-03T19:00:56.308Z INFO (Dispatcher) Set running task {"task": {"task_name":"backup_hourly","status":"waiting","start":"2024-01-03T19:00:56.285213118Z","error_code":0,"type":"BACKUP"}} 2024-01-03T19:00:56.309Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr backup --cluster http://127.0.0.1:8091 --archive /home/ec2-user/cb_backup_20240101 --repo 023110cf-78f9-494d-a023-982356cc6260 --purge --auto-select-threads --json-progress 31d077db-c79f-4baf-9b10-b7f703833684 2024-01-03T19:00:56.311Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T19:00:56.311Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "waiting"} 2024-01-03T19:00:56.313Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T19:00:56.313Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "running"} 2024-01-03T19:00:56.330Z INFO (Dispatcher) Task dispatched {"nodeID": "80802b2860060fffd26a01149967b587"} 2024-01-03T19:00:56.348Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T19:00:56.348Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T19:01:05.000Z INFO (Worker) Command exited {"err": null, "cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T19:01:05.000Z INFO (Worker) Task done {"cluster": "self", "repositoryID": "repo_jan3_07_45_utc", "state": "active", "taskName": "backup_hourly"} 2024-01-03T19:01:05.003Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T19:01:05.003Z DEBUG (Event Handler) (Node Run) Task updates {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T19:01:05.003Z INFO (Event Handler) (Node Run) Task finished {"cluster": "self", "repository": "repo_jan3_07_45_utc", "task": "backup_hourly", "status": "done"} 2024-01-03T19:01:05.040Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T19:01:05.040Z DEBUG (Leader Manager) Received store event {"eventType": 5} 2024-01-03T19:01:05.040Z DEBUG (Leader Manager) Received store event {"eventType": 3} 2024-01-03T19:03:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T19:03:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:03:58.950Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:08:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T19:08:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:08:58.945Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:13:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T19:13:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:13:58.971Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:18:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T19:18:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:18:58.965Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:23:58.760Z INFO (Stats) Start repositories data size collection 2024-01-03T19:23:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:23:58.979Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:28:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T19:28:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:28:58.968Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:33:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T19:33:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:33:58.934Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:38:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T19:38:58.763Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:38:58.938Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:43:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T19:43:58.758Z INFO (Janitor) Running orphan task detection 2024-01-03T19:43:58.759Z INFO (Janitor) Running storage access check 2024-01-03T19:43:58.766Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:43:58.774Z INFO (Janitor) Running storage access check on state active 2024-01-03T19:43:58.783Z INFO (GRPC Manager) Global storage location check {"req": "location:\"/home/ec2-user/cb_backup_20240101/.cbbs-0a00a7717f426e18e6cd4bd07e79f0e7/node-80802b2860060fffd26a01149967b587-1704311038776923871\" sourceNode:\"80802b2860060fffd26a01149967b587\""} 2024-01-03T19:43:58.786Z INFO (Janitor) Running storage access check on state archived 2024-01-03T19:43:58.977Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:48:58.759Z INFO (Stats) Start repositories data size collection 2024-01-03T19:48:58.764Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:48:58.963Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:48:58.970Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:49:01.714Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 10 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:53:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T19:53:58.771Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:53:59.050Z INFO (Stats) Stop repositories data size collection 2024-01-03T19:58:58.758Z INFO (Stats) Start repositories data size collection 2024-01-03T19:58:58.765Z INFO (Runner) Running command /opt/couchbase/bin/cbbackupmgr info --archive /home/ec2-user/cb_backup_20240101 --offset 0 --limit 0 --json --repo 023110cf-78f9-494d-a023-982356cc6260 2024-01-03T19:58:59.003Z INFO (Stats) Stop repositories data size collection