Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Minor
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: analytics
Labels:
Environment:
Upgrade from 6.6.2-9588 to 7.0.0-4939

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
cbcollect logs:

url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.15.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.214.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.244.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.245.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.102.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.109.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.118.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.164.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.206.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.210.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.29.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.61.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.90.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.93.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.117.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.191.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.207.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.209.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.225.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.54.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.70.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.110.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.110.76.zip

Show
cbcollect logs: url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.15.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.214.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.232.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.244.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.104.245.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.102.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.109.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.118.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.164.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.206.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.210.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.29.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.61.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.90.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.105.93.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.117.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.191.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.207.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.209.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.225.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.54.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.106.70.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.110.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/MB-45646/collectinfo-2021-04-14T095842-ns_1%40172.23.110.76.zip
Story Points:
1
Is this a Regression?:
No
Sprint:
CX Sprint 245

Description

Steps to repro
1. Ran the longevity test we had on 6.6.2-9588 - 30 node cluster
2. Start data load on all buckets
3. Started failover and upgrade(to 7.0.0-4939) and rebalance of 172.23.105.61 172.23.106.209 172.23.106.32 172.23.106.70 172.23.110.75 172.23.110.76
4. Rebalance completes successfully.

However I see lot of messages like below on the UI

172.23.110.75

Service 'cbas' exited with status 1. Restarting. Messages:

2021-04-14T02:39:40.490-07:00 CRIT CBAS.cbas my node is in topology but not registered in cbauth db

2021-04-14T02:39:40.490-07:00 WARN CBAS.cbas driver bootstrap attempt failed: my node is in topology but not registered in cbauth db

2021-04-14T02:39:40.490-07:00 INFO CBAS.cbas driver bootstrap failed; will retry in 1s (2 retries remaining)

2021-04-14T02:39:41.490-07:00 CRIT CBAS.cbas my node is in topology but not registered in cbauth db

2021-04-14T02:39:41.490-07:00 WARN CBAS.cbas driver bootstrap attempt failed: my node is in topology but not registered in cbauth db

2021-04-14T02:39:41.490-07:00 INFO CBAS.cbas driver bootstrap failed; will retry in 1s (1 retries remaining)

2021-04-14T02:39:42.490-07:00 CRIT CBAS.cbas my node is in topology but not registered in cbauth db

2021-04-14T02:39:42.490-07:00 WARN CBAS.cbas driver bootstrap attempt failed: my node is in topology but not registered in cbauth db

2021-04-14T02:39:42.490-07:00 FATA CBAS.cbas unable to complete driver bootstrap after 60 attempt(s) (last failure: my node is in topology but not registered in cbauth db)

172.23.110.75 :

/opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:06:07.005-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:07:06.299-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:08:05.556-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:09:04.830-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:10:04.089-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:11:03.347-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:12:02.633-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:13:01.918-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:14:01.167-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:15:00.467-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:15:59.734-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:16:58.990-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:17:58.270-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:18:57.523-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:19:56.820-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:20:56.109-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:21:55.380-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:22:54.635-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:23:53.901-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:24:53.172-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:25:52.441-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:26:51.716-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:27:50.978-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:28:50.255-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:29:49.555-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:30:48.855-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:31:48.164-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:32:47.439-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:33:46.746-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:34:46.022-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:35:45.289-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:36:44.559-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:37:43.870-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:38:43.177-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:39:42.493-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:40:41.785-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:41:41.094-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:42:40.368-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:43:39.676-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:44:38.961-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages:

Also noticed following exceptions on another analytics node.
172.23.105.90

2021-04-10T04:28:44.920-07:00 FATA CBAS.nc.CcConnection [Executor-57:239e6360311cc285ccf15de3cc3556f1] Registering with ClusterControllerRemoteProxy [/172.23.105.210:9112] failed with exception

2021-04-10T04:28:45.060-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] JVM halting with status 15; thread dump at halt:

2021-04-11T05:51:12.944-07:00 FATA CBAS.nc.CcConnection [Executor-1534:c46a2e4dd26637eccaea9ab5969abaf1] Registering with ClusterControllerRemoteProxy [/172.23.105.210:9112] failed with exception

2021-04-11T05:51:13.208-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] JVM halting with status 15; thread dump at halt:

2021-04-12T09:28:29.305-07:00 FATA CBAS.nc.CcConnection [Executor-36:f7062dd29cb2866a3da899bcf93d1a74] Registering with ClusterControllerRemoteProxy [/172.23.105.210:9112] failed with exception

2021-04-12T09:28:29.584-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] JVM halting with status 15; thread dump at halt:

cbcollect_info attached.

Attachments

Issue Links

backports to

MB-46297 [BP 6.6.3] "Service 'cbas' exited with status 1" during online upgrade with failover

Closed

is duplicated by

MB-45914 [Upgrade] - Service 'cbas' exited with status 1. Restarting. seen during upgrade from 6.6.2 -> 7.0.0

Closed

Activity

People

Assignee:: Michael Blow

Reporter:: Balakumaran Gopal

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 14/Apr/21 3:21 AM

Updated:: 17/Jun/21 3:17 PM

Resolved:: 30/Apr/21 7:26 PM