Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
Columnar 1.0.0
-
2239
-
Untriaged
-
0
-
Unknown
Description
- Create 32 node columnar cluster and 20 kafka collections. Let the ingestion complete
- Disconnect previous link and create a new link.
- Create 20 new collections on new link and perform scaling 32 -> 16 -> 8 -> 4 -> 8 -> 16 -> 32
- Disconnect previous link and create a new link.
- Create 20 new collections on new link and perform scaling 32 -> 16 -> 8 -> 4 -> 8 -> 16 -> 32
- Disconnect previous link and that fails because cluster is in recovering state.
Exception: Status: 503, content: {u'errors': [{u'msg': u'Analytics Service is temporarily unavailable', u'code': 23000, u'retriable': True}], u'status': u'errors'}, Errors: [{u'msg': u'Analytics Service is temporarily unavailable', u'code': 23000, u'retriable': True}]
- Prior to that below message is observed and cluster goes into UNUSABLE state
1351 2024-07-26T23:47:09.528+00:00 INFO CBAS.adapter.TopicRecordReader [Executor-199:fa0281a3fd52b0f749cee71a168493b3] Error while polling data from Kafka1352 org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException1353 at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.maybeThrowInterruptException(ConsumerNetworkClient.java:535) ~[kafka-clients-3.6.0.jar:?]1354 at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:296) ~[kafka-clients-3.6.0.jar:?]
1355 at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:251) ~[kafka-clients-3.6.0.jar:?]1356 at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1255) ~[kafka-clients-3.6.0.jar:?]1357 at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1186) ~[kafka-clients-3.6.0.jar:?]1358 at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159) ~[kafka-clients-3.6.0.jar:?]
1359 at com.couchbase.analytics.adapter.TopicRecordReader.dataPoll(TopicRecordReader.java:370) [columnar-connector.jar:1.0.0-2239]1360 at com.couchbase.analytics.adapter.TopicRecordReader$KafkaTopicDataPoller.run(TopicRecordReader.java:398) [columnar-connector.jar:1.0.0-2239]1361 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]1362 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
1363 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]1364 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]1365 at java.base/java.lang.Thread.run(Thread.java:840) [?:?]1366 Caused by: java.lang.InterruptedException
1367 ... 13 more
Why did the cluster goes into UNUSABLE state so that it has to RECOVER and during that phase no query is accepted by the service?