logs_node (backup_service.log): ------------------------------- 2021-04-21T09:57:34.782-0500 INFO (Main) Running node version backup-0.0.0-0000- with options: -http-port=7100 -grpc-port=7200 -https-port=17100 -cert-path=/Users/rebeccacritchlow/Documents/test-source/ns_server/data/n_0/config/memcached-cert.pem -key-path=/Users/rebeccacritchlow/Documents/test-source/ns_server/data/n_0/config/memcached-key.pem -cbm=/Users/rebeccacritchlow/Documents/test-source/install/bin/cbbackupmgr -node-uuid=8b85afc04d06610ec74352f47ba7c05b -public-address=127.0.0.1 -admin-port=9000 -log-file=none -log-level=debug -integrated-mode -integrated-mode-host=http://127.0.0.1:9000 -secure-integrated-mode-host=https://127.0.0.1:19000 -integrated-mode-user=@backup -default-collect-logs-path=/Users/rebeccacritchlow/Documents/test-source/ns_server/tmp -cbauth-host=127.0.0.1:9000 2021-04-21T09:57:34.783-0500 INFO (Main) Initialized logger {"log level": "debug"} 2021-04-21T09:57:34.783-0500 INFO (Main) Getting credentials 2021-04-21T09:57:34.783-0500 DEBUG (Main) File limit info {"curr": 4096, "max": 4096, "err": null} 2021-04-21T09:57:34.786-0500 DEBUG (Rest) GET http://127.0.0.1:9000/pools 200 2021-04-21T09:57:34.787-0500 INFO (Node) Confirmed cluster uuid {"clusterUUID": "4beb9611b9fbacc4b01140bb5d4a49d5"} 2021-04-21T09:57:34.812-0500 INFO (CBAUDIT) audit: created new audit service 2021-04-21T09:57:34.812-0500 INFO (Audit) Audit service initialised 2021-04-21T09:57:34.814-0500 INFO (Node) Starting GRPC manager 2021-04-21T09:57:34.814-0500 INFO (GRPC Manager) Starting GRPC Server {"address": "[::]:7200"} 2021-04-21T09:57:34.816-0500 INFO (Node) Starting leader manager 2021-04-21T09:57:34.816-0500 INFO (Leader Manager) Starting leader manager 2021-04-21T09:57:34.816-0500 INFO (HTTP Manager) HTTP server started on {"address": ":7100"} 2021-04-21T09:57:34.816-0500 INFO (HTTP Manager) HTTPS server started on {"address": ":17100"} 2021-04-21T09:57:34.818-0500 INFO (Storage) Starting MetaKV observer 2021-04-21T09:57:34.818-0500 INFO (Node) TLS config refreshed 2021-04-21T09:57:34.818-0500 WARN (Worker) Stopping task runner with 0 task on queue 2021-04-21T09:57:34.818-0500 WARN (Worker) Stopped 2021-04-21T09:57:34.819-0500 WARN (HTTP Manager) HTTPS server stopped {"err": "http: Server closed"} 2021-04-21T09:57:34.819-0500 INFO (HTTP Manager) Shutting down http server 2021-04-21T09:57:34.819-0500 WARN (HTTP Manager) HTTP server stopped {"err": "http: Server closed"} 2021-04-21T09:57:34.819-0500 DEBUG (HTTP Manager) HTTP servers shutdown {"time_taken": 0.000246648} 2021-04-21T09:57:34.819-0500 INFO (GRPC Manager) Starting GRPC Server {"address": "[::]:7200"} 2021-04-21T09:57:34.821-0500 DEBUG (Leader Manager) Received store event {"eventType": 6} 2021-04-21T09:57:34.822-0500 DEBUG (Leader Manager) Received store event {"eventType": 6} 2021-04-21T09:57:34.824-0500 INFO (HTTP Manager) HTTPS server started on {"address": ":17100"} 2021-04-21T09:57:34.824-0500 INFO (HTTP Manager) HTTP server started on {"address": ":7100"} 2021-04-21T09:57:36.061-0500 INFO (Service Manager) Prepare topology change {"change": {"id":"be05f334290eeecbbe86f5b0d69db771","currentTopologyRev":null,"type":"topology-change-rebalance","keepNodes":[{"nodeInfo":{"nodeId":"8b85afc04d06610ec74352f47ba7c05b","priority":1,"opaque":{"grpc_port":7200,"host":"127.0.0.1","http_port":7100}},"recoveryType":"recovery-full"}],"ejectNodes":[]}} 2021-04-21T09:57:36.064-0500 INFO (Service Manager) Start topology change {"change": {"id":"be05f334290eeecbbe86f5b0d69db771","currentTopologyRev":null,"type":"topology-change-rebalance","keepNodes":[{"nodeInfo":{"nodeId":"8b85afc04d06610ec74352f47ba7c05b","priority":1,"opaque":{"grpc_port":7200,"host":"127.0.0.1","http_port":7100}},"recoveryType":"recovery-full"}],"ejectNodes":[]}} 2021-04-21T09:57:36.064-0500 INFO (Rebalance) Starting rebalance {"change": {"id":"be05f334290eeecbbe86f5b0d69db771","currentTopologyRev":null,"type":"topology-change-rebalance","keepNodes":[{"nodeInfo":{"nodeId":"8b85afc04d06610ec74352f47ba7c05b","priority":1,"opaque":{"grpc_port":7200,"host":"127.0.0.1","http_port":7100}},"recoveryType":"recovery-full"}],"ejectNodes":[]}} 2021-04-21T09:57:36.073-0500 INFO (Rebalance) Got old leader {"leader": ""} 2021-04-21T09:57:36.077-0500 INFO (Rebalance) Got current nodes {"#nodes": 0} 2021-04-21T09:57:36.077-0500 INFO (Rebalance) Setting self as leader 2021-04-21T09:57:36.079-0500 DEBUG (Leader Manager) Received store event {"eventType": 1} 2021-04-21T09:57:36.079-0500 INFO (Leader Manager) Stepped up as leader 2021-04-21T09:57:36.079-0500 INFO (Leader Manager) Leader value changed {"leader": true} 2021-04-21T09:57:36.079-0500 INFO (Janitor) Starting janitor 2021-04-21T09:57:36.079-0500 INFO (Stats) Starting periodic stat collection {"period": "5m0s"} 2021-04-21T09:57:36.079-0500 INFO (Rebalance) Did the failover nodes will do eject node now {"eject nodes": []} 2021-04-21T09:57:36.079-0500 INFO (Janitor) Starting time jump checker 2021-04-21T09:57:36.079-0500 INFO (Rebalance) Adding nodes now {"keep nodes": [{"nodeInfo":{"nodeId":"8b85afc04d06610ec74352f47ba7c05b","priority":1,"opaque":{"grpc_port":7200,"host":"127.0.0.1","http_port":7100}},"recoveryType":"recovery-full"}]} 2021-04-21T09:57:36.079-0500 INFO (Rebalance) Keep node {"nodeID": "8b85afc04d06610ec74352f47ba7c05b"} 2021-04-21T09:57:36.079-0500 INFO (Leader Manager) Constructing schedule from storage 2021-04-21T09:57:36.080-0500 DEBUG (Rebalance) Added node to MetaKV {"nodeID": "8b85afc04d06610ec74352f47ba7c05b"} 2021-04-21T09:57:36.082-0500 DEBUG (Rebalance) Checking if status match {"expected": "node in", "attempt": 0} 2021-04-21T09:57:36.083-0500 DEBUG (Leader Manager) Received store event {"eventType": 2} 2021-04-21T09:57:36.083-0500 INFO (Event Handler) (Topology) Node has been added to the cluster 2021-04-21T09:57:36.083-0500 INFO (Leader Manager) Switch status {"nodeStatus": "node in"} 2021-04-21T09:57:41.086-0500 DEBUG (Rebalance) Checking if status match {"expected": "node in", "attempt": 1} 2021-04-21T09:57:41.086-0500 INFO (Rebalance) Nodes added 2021-04-21T09:57:41.086-0500 INFO (ClockKeeper) Starting scheduler 2021-04-21T09:57:41.086-0500 INFO (Rebalance) Rebalance done {"err": null, "state": {}, "cancelled": false} 2021-04-21T09:57:41.086-0500 INFO (Janitor) Storage access check triggered 2021-04-21T09:57:41.086-0500 INFO (Janitor) Running storage access check 2021-04-21T09:57:41.089-0500 INFO (Janitor) Running storage access check on state active 2021-04-21T09:57:41.091-0500 INFO (Janitor) Running storage access check on state archived 2021-04-21T09:59:02.727-0500 INFO (REST) POST /api/v1/cluster/self/repository/active/simple 2021-04-21T09:59:18.923-0500 INFO (REST) POST /api/v1/cluster/self/repository/active/simple 2021-04-21T09:59:22.860-0500 INFO (REST) POST /api/v1/cluster/self/repository/active/simple 2021-04-21T09:59:22.875-0500 INFO (GRPC Manager) Global storage location check {"req": "location:\"/Users/rebeccacritchlow/data/backups/.cbbs-4beb9611b9fbacc4b01140bb5d4a49d5/node-8b85afc04d06610ec74352f47ba7c05b-1619017162872257000\" sourceNode:\"8b85afc04d06610ec74352f47ba7c05b\""} 2021-04-21T09:59:22.880-0500 INFO (Runner) Running command /Users/rebeccacritchlow/Documents/test-source/install/bin/cbbackupmgr config -a /Users/rebeccacritchlow/data/backups -r b253df55-e8c6-4501-9c20-e0980e9783b6 --readme-author Backup Service, cluster: 4beb9611b9fbacc4b01140bb5d4a49d5 --cbbs 2021-04-21T09:59:24.641-0500 DEBUG (Leader Manager) Received store event {"eventType": 4} 2021-04-21T09:59:24.641-0500 INFO (Event Handler) (Repo Change) Repository change {"event": {"Type":4,"Path":"/cbbs/cluster/self/active/simple","Value":"eyJpZCI6InNpbXBsZSIsInBsYW5fbmFtZSI6Il9ob3VybHlfYmFja3VwcyIsInN0YXRlIjoiYWN0aXZlIiwiYXJjaGl2ZSI6Ii9Vc2Vycy9yZWJlY2NhY3JpdGNobG93L2RhdGEvYmFja3VwcyIsInJlcG8iOiJiMjUzZGY1NS1lOGM2LTQ1MDEtOWMyMC1lMDk4MGU5NzgzYjYiLCJ2ZXJzaW9uIjoxLCJjcmVhdGlvbl90aW1lIjoiMjAyMS0wNC0yMVQwOTo1OToyNC42MzUzMjYtMDU6MDAiLCJ1cGRhdGVfdGltZSI6IjIwMjEtMDQtMjFUMDk6NTk6MjQuNjM1MzI2LTA1OjAwIn0=","Rev":"g2wAAAABaAJtAAAAIDhiODVhZmMwNGQwNjYxMGVjNzQzNTJmNDdiYTdjMDViaAJhAW4FAMy59NkOag=="}} 2021-04-21T09:59:24.641-0500 INFO (Event Handler) (Repo Change) Resuming repository {"repositoryID": "simple", "planName": "_hourly_backups"} 2021-04-21T09:59:24.643-0500 INFO (Manger) Added repository {"id": {"id":"simple","plan_name":"_hourly_backups","state":"active","archive":"/Users/rebeccacritchlow/data/backups","repo":"b253df55-e8c6-4501-9c20-e0980e9783b6","version":1,"creation_time":"0001-01-01T00:00:00Z","update_time":"0001-01-01T00:00:00Z"}, "cluster": "self", "archive": "/Users/rebeccacritchlow/data/backups", "repo": "b253df55-e8c6-4501-9c20-e0980e9783b6", "plan": "_hourly_backups"} 2021-04-21T09:59:24.644-0500 INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "simple", "task": "backup_hourly"} 2021-04-21T09:59:24.679-0500 INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "simple", "task": "merge_every_6_hours"} 2021-04-21T09:59:24.695-0500 INFO (ClockKeeper) Adding job {"cluster": "self", "repository": "simple", "task": "merge_week"} -------------------------------