Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46297

[BP 6.6.3] "Service 'cbas' exited with status 1" during online upgrade with failover

    XMLWordPrintable

Details

    Description

      Steps to repro
      1. Ran the longevity test we had on 6.6.2-9588 - 30 node cluster
      2. Start data load on all buckets
      3. Started failover and upgrade(to 7.0.0-4939) and rebalance of 172.23.105.61 172.23.106.209 172.23.106.32 172.23.106.70 172.23.110.75 172.23.110.76
      4. Rebalance completes successfully.

      However I see lot of messages like below on the UI

      172.23.110.75

      Service 'cbas' exited with status 1. Restarting. Messages:
      		 
      2021-04-14T02:39:40.490-07:00 CRIT CBAS.cbas my node is in topology but not registered in cbauth db
      		 
      2021-04-14T02:39:40.490-07:00 WARN CBAS.cbas driver bootstrap attempt failed: my node is in topology but not registered in cbauth db
      		 
      2021-04-14T02:39:40.490-07:00 INFO CBAS.cbas driver bootstrap failed; will retry in 1s (2 retries remaining)
      		 
      2021-04-14T02:39:41.490-07:00 CRIT CBAS.cbas my node is in topology but not registered in cbauth db
      		 
      2021-04-14T02:39:41.490-07:00 WARN CBAS.cbas driver bootstrap attempt failed: my node is in topology but not registered in cbauth db
      		 
      2021-04-14T02:39:41.490-07:00 INFO CBAS.cbas driver bootstrap failed; will retry in 1s (1 retries remaining)
      		 
      2021-04-14T02:39:42.490-07:00 CRIT CBAS.cbas my node is in topology but not registered in cbauth db
      		 
      2021-04-14T02:39:42.490-07:00 WARN CBAS.cbas driver bootstrap attempt failed: my node is in topology but not registered in cbauth db
      		 
      2021-04-14T02:39:42.490-07:00 FATA CBAS.cbas unable to complete driver bootstrap after 60 attempt(s) (last failure: my node is in topology but not registered in cbauth db)

       
      172.23.110.75 : 

      /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:06:07.005-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:07:06.299-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:08:05.556-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:09:04.830-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:10:04.089-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:11:03.347-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:12:02.633-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:13:01.918-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:14:01.167-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:15:00.467-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:15:59.734-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:16:58.990-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:17:58.270-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:18:57.523-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:19:56.820-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:20:56.109-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:21:55.380-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:22:54.635-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:23:53.901-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:24:53.172-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:25:52.441-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:26:51.716-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:27:50.978-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:28:50.255-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:29:49.555-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:30:48.855-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:31:48.164-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:32:47.439-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:33:46.746-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:34:46.022-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:35:45.289-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:36:44.559-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:37:43.870-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:38:43.177-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:39:42.493-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:40:41.785-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:41:41.094-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:42:40.368-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:43:39.676-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages: /opt/couchbase/var/lib/couchbase/logs/info.log:[user:info,2021-04-14T02:44:38.961-07:00,ns_1@172.23.110.75:<0.609.0>:ns_log:crash_consumption_loop:63]Service 'cbas' exited with status 1. Restarting. Messages:

       

      Also noticed following exceptions on another analytics node.
      172.23.105.90

      2021-04-10T04:28:44.920-07:00 FATA CBAS.nc.CcConnection [Executor-57:239e6360311cc285ccf15de3cc3556f1] Registering with ClusterControllerRemoteProxy [/172.23.105.210:9112] failed with exception
      		 
      2021-04-10T04:28:45.060-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] JVM halting with status 15; thread dump at halt:
      		 
      2021-04-11T05:51:12.944-07:00 FATA CBAS.nc.CcConnection [Executor-1534:c46a2e4dd26637eccaea9ab5969abaf1] Registering with ClusterControllerRemoteProxy [/172.23.105.210:9112] failed with exception
      		 
      2021-04-11T05:51:13.208-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] JVM halting with status 15; thread dump at halt:
      		 
      2021-04-12T09:28:29.305-07:00 FATA CBAS.nc.CcConnection [Executor-36:f7062dd29cb2866a3da899bcf93d1a74] Registering with ClusterControllerRemoteProxy [/172.23.105.210:9112] failed with exception
      		 
      2021-04-12T09:28:29.584-07:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] JVM halting with status 15; thread dump at halt:

      cbcollect_info attached. 

      Attachments

        Issue Links

          is a backport of

          Bug - A problem which impairs or prevents the functions of the product. MB-45648 "Service 'cbas' exited with status 1" during online upgrade with failover

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-46781 [BP 6.6.3][system test upgrade] : Analytics rebalance fails with "java.lang.IllegalStateException: timed out waiting for all nodes to join & cluster active" during upgrade from 6.6.2 -> 7.0.0

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              michael.blow Michael Blow
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty