Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
5.5.0
-
5.5.0-2891
-
Untriaged
-
-
Yes
-
CX Sprint 106, CX Sprint 107
Description
1. 1KV and 2 cbas nodes in the clsuter.
2. Do a swap rebalance of cc node with a new node. Rebalance is successful but Analytics cluster is down.
UI:
Service 'cbas' exited with status 1. Restarting. Messages:
2018-06-12T23:45:18.361-07:00 INFO CBAS.bootstrap.NCApplication [ShutdownHook-c1a4f790ace4f10e9041563ac23e7530] Stopping Asterix node controller: c1a4f790ace4f10e9041563ac23e7530
2018-06-12T23:45:18.561-07:00 INFO CBAS.nc.ReplicaManager [HttpExecutor(port:9110)-0] ignoring request to add replica to ourselves
2018-06-12T23:45:18.575-07:00 INFO CBAS.cbas Adding replica 8da78bf995e3f08575061c6cd33b557a at 0.0.0.0:9120
2018-06-12T23:45:18.588-07:00 WARN CBAS.server.HttpServerHandler [nioEventLoopGroup-7-1] Request rejected by server executor service. Task java.util.concurrent.FutureTask@37b8adec rejected from java.util.concurrent.ThreadPoolExecutor@5f7d6de3[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
2018-06-12T23:45:18.589-07:00 ERRO CBAS.cbas Failed to add replica Unexpected response 503
2018-06-12T23:45:18.589-07:00 FATA CBAS.cbas Unexpected failure while sync'ing replicas Unexpected response 503
[goport(/opt/couchbase/bin/cbas)] 2018/06/12 23:45:18 child process exited with status 1
Logs:
2018-06-12T23:45:11.554-07:00 ERRO CBAS.message.RegistrationTasksResponseMessage [Executor-9:c1a4f790ace4f10e9041563ac23e7530] Failed during startup task
|
org.apache.hyracks.api.exceptions.HyracksDataException: org.apache.asterix.common.exceptions.MetadataException: Could not find required metadata indexes. Please delete /opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0/txn-log to intialize as a new instance. (WARNING: all data will be lost.)
|
at org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:51) ~[hyracks-api.jar:5.5.0-2891]
|
at org.apache.asterix.app.nc.task.MetadataBootstrapTask.perform(MetadataBootstrapTask.java:45) ~[asterix-app.jar:5.5.0-2891]
|
at org.apache.asterix.app.replication.message.RegistrationTasksResponseMessage.handle(RegistrationTasksResponseMessage.java:61) ~[asterix-app.jar:5.5.0-2891]
|
at org.apache.asterix.messaging.NCMessageBroker.lambda$receivedMessage$0(NCMessageBroker.java:100) ~[asterix-app.jar:5.5.0-2891]
|
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_162]
|
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_162]
|
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
|
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
|
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
|
Caused by: org.apache.asterix.common.exceptions.MetadataException: Could not find required metadata indexes. Please delete /opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0/txn-log to intialize as a new instance. (WARNING: all data will be lost.)
|
at org.apache.asterix.metadata.bootstrap.MetadataBootstrap.startUniverse(MetadataBootstrap.java:181) ~[asterix-metadata.jar:5.5.0-2891]
|
at org.apache.asterix.app.nc.NCAppRuntimeContext.initializeMetadata(NCAppRuntimeContext.java:426) ~[asterix-app.jar:5.5.0-2891]
|
at com.couchbase.analytics.bootstrap.AnalyticsNcApplicationContext.initializeMetadata(AnalyticsNcApplicationContext.java:215) ~[cbas-server.jar:5.5.0-2891]
|
at org.apache.asterix.app.nc.task.MetadataBootstrapTask.perform(MetadataBootstrapTask.java:43) ~[asterix-app.jar:5.5.0-2891]
|
... 7 more
|
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Could not find required metadata indexes. Please delete /opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0/txn-log to intialize as a new instance. (WARNING: all data will be lost.)
|
at org.apache.asterix.metadata.bootstrap.MetadataBootstrap.enlistMetadataDataset(MetadataBootstrap.java:355) ~[asterix-metadata.jar:5.5.0-2891]
|
at org.apache.asterix.metadata.bootstrap.MetadataBootstrap.startUniverse(MetadataBootstrap.java:148) ~[asterix-metadata.jar:5.5.0-2891]
|
at org.apache.asterix.app.nc.NCAppRuntimeContext.initializeMetadata(NCAppRuntimeContext.java:426) ~[asterix-app.jar:5.5.0-2891]
|
at com.couchbase.analytics.bootstrap.AnalyticsNcApplicationContext.initializeMetadata(AnalyticsNcApplicationContext.java:215) ~[cbas-server.jar:5.5.0-2891]
|
at org.apache.asterix.app.nc.task.MetadataBootstrapTask.perform(MetadataBootstrapTask.java:43) ~[asterix-app.jar:5.5.0-2891]
|
... 7 more
|
2018-06-12T23:45:11.565-07:00 INFO CBAS.work.WorkQueue [Worker:ClusterController] Executing: ApplicationMessage: nodeID: c1a4f790ace4f10e9041563ac23e7530
|
2018-06-12T23:45:11.565-07:00 INFO CBAS.util.ExitUtil [JVM exit thread] JVM exiting with status 2; bye!
|
Attachments
Issue Links
- links to