Details
-
Bug
-
Resolution: Fixed
-
Major
-
Columnar 1.0.0
-
1.0.0-2230
-
Untriaged
-
0
-
Unknown
Description
On scale up from 8 to 16 nodes, the new nodes(009 to 016) have reported cbas exit 1 with the following error -
2024-07-22T20:22:07.861+00:00 WARN CBAS.cbas got error obtaining current special user creds from cbauth: Unable to find given hostport in cbauth database: `svc-da-node-016.8cc-phsibj7kjt6w.sandbox.nonprod-project-avengers.com:8095'; will retry |
2024-07-22T20:22:08.540+00:00 INFO CBAS.cbas Initialized starting partitionId to 0 for node fdbb9f48cf6c714c6c26366a00e2c346 |
2024-07-22T20:22:08.540+00:00 INFO CBAS.cbas Initialized controllerId to 0 for node fdbb9f48cf6c714c6c26366a00e2c346 |
2024-07-22T20:22:08.540+00:00 INFO CBAS.cbas no CC / Metadata node defined in metakv- will pick *this* node, fdbb9f48cf6c714c6c26366a00e2c346 |
2024-07-22T20:22:08.541+00:00 INFO CBAS.cbas our node is within topology refresh bf86ce7731858b3711787c8077090a49/1; ensuring driver is running... |
2024-07-22T20:22:08.544+00:00 INFO CBAS.cbas initializing audit service at: http://127.0.0.1:8091 |
2024-07-22T20:22:08.553+00:00 INFO CBAS.cbas audit: created new audit service |
2024-07-22T20:22:08.553+00:00 INFO CBAS.cbas ensure blob storage settings failed; will retry every 500ms (for up to 600 attempts) |
2024-07-22T20:27:08.736+00:00 ERRO CBAS.cbas cbas process aborting with exit code 1 due to unable to complete ensure blob storage settings after 600 attempt(s) (last failure: blob storage scheme has invalid value) |
2024-07-22T20:27:08.736+00:00 FATA CBAS.cbas unable to complete ensure blob storage settings after 600 attempt(s) (last failure: blob storage scheme has invalid value) |
2024-07-22T20:27:08.881+00:00 WARN CBAS.cbas Updating log level to DEBUG (1) |
2024-07-22T20:27:08.881+00:00 INFO CBAS.cbas ++++++++++++ starting cbas main ++++++++++++ |
2024-07-22T20:27:08.881+00:00 INFO CBAS.cbas deployment model: (columnar) |
2024-07-22T20:27:08.881+00:00 INFO CBAS.cbas uuid: fdbb9f48cf6c714c6c26366a00e2c346 |
2024-07-22T20:27:08.881+00:00 INFO CBAS.cbas IPv4 addresses: [127.0.0.1 10.0.2.63 10.0.2.51] |
2024-07-22T20:27:08.881+00:00 INFO CBAS.cbas IPv6 addresses: [::1%lo fe80::8ff:f5ff:fe51:1b89%eth0 fe80::8ff:d0ff:fe5e:6efd%eth1] |
2024-07-22T20:27:08.881+00:00 INFO CBAS.cbas setting ipv4 required, ipv6 off |
2024-07-22T20:27:08.882+00:00 INFO CBAS.cbas allocating 18846MiB/37693MiB for heap min/max (reserved 12564MiB of 50257MiB service quota for non-heap allocations) |
2024-07-22T20:27:08.882+00:00 INFO CBAS.cbas a single io device was provided; auto-adjusting... |
2024-07-22T20:27:08.882+00:00 INFO CBAS.cbas configuring io devices based on logical CPUs: 16 |
2024-07-22T20:27:08.882+00:00 INFO CBAS.cbas io devices were adjusted to a total of: 16 |
Note - This doesn't occur all the time. Other scaling operations ( 4 to 8 or 16 to 32) haven't seen this. There is an old closed bug that looks similar. There's something specific about this rebalance that has caused this.
cbcollect ->
Attachments
Issue Links
- relates to
-
AV-76982 Loading...