Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53739

IPV6: Backup service is crashing continuously with exit status 1

    XMLWordPrintable

Details

    Description

      Build: 7.1.2-3432

      Scenario:
      1. Enable ipv6-only on s22401

      2022-09-18 05:32:43 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on s22401-ip6.qe.couchbase.com: /opt/couchbase/bin/couchbase-cli ip-family  -c s22401-ip6.qe.couchbase.com:8091  -u Administrator -p password --set  --ipv6only 
      2022-09-18 05:32:44 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully with root
      2022-09-18 05:32:44 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Switched IP family for node: http://s22401-ip6.qe.couchbase.com:8091
      2022-09-18 05:32:44 | INFO | MainProcess | test_thread | [remote_util.log_command_output] SUCCESS: Switched IP family of the cluster
      

      2. Add second node (s22404) and rebalance the cluster.

      Observing the rebalance fails with backup service crashing with status code 1.

      {service_rebalance_failed,backup,
      {agent_died,<0.10116.0>,
      {linked_process_died,<0.10117.0>,
      {'ns_1@s22401-ip6.qe.couchbase.com', {no_connection,\"backup-service_api\"}}}}}."}

      From debug.log

      =========================WARNING REPORT=========================
      Description: "Authenticity is not established by certificate path validation"
           Reason: "Option {verify, verify_peer} and cacertfile/cacerts is missing"[ns_server:debug,2022-09-20T02:28:45.690-07:00,ns_1@s22401-ip6.qe.couchbase.com:json_rpc_connection-backup-cbauth<0.4988.0>:json_rpc_connection:init:68]Observed revrpc connection: label "backup-cbauth", handling process <0.4988.0>
      [ns_server:debug,2022-09-20T02:28:45.690-07:00,ns_1@s22401-ip6.qe.couchbase.com:menelaus_cbauth<0.552.0>:menelaus_cbauth:handle_cast:101]Observed json rpc process {"backup-cbauth",<0.4988.0>} started
      [ns_server:debug,2022-09-20T02:28:45.728-07:00,ns_1@s22401-ip6.qe.couchbase.com:json_rpc_connection-backup-cbauth<0.4988.0>:json_rpc_connection:handle_info:124]Socket closed
      [ns_server:debug,2022-09-20T02:28:45.729-07:00,ns_1@s22401-ip6.qe.couchbase.com:menelaus_cbauth<0.552.0>:menelaus_cbauth:handle_info:136]Observed json rpc process {"backup-cbauth",<0.4988.0>} died with reason shutdown
      [user:info,2022-09-20T02:28:45.729-07:00,ns_1@s22401-ip6.qe.couchbase.com:<0.404.0>:ns_log:consume_log:76]Service 'backup' exited with status 1. Restarting. Messages:
      2022-09-20T02:28:45.722-07:00 INFO (Main) Initialized logger {"log level": "debug"}
      2022-09-20T02:28:45.722-07:00 INFO (Main) Getting credentials
      2022-09-20T02:28:45.722-07:00 DEBUG (Main) File limit info {"curr": 200000, "max": 200000, "err": null}
      2022-09-20T02:28:45.722-07:00 DEBUG (REST) (Attempt 1) (GET) Dispatching request to 'http://127.0.0.1:8091/pools/default/nodeServices'
      2022-09-20T02:28:45.724-07:00 ERROR (REST) (Attempt 1) (GET) Failed to perform request to 'http://127.0.0.1:8091/pools/default/nodeServices': Get "http://127.0.0.1:8091/pools/default/nodeServices": dial tcp 127.0.0.1:8091: connect: connection refused
      2022-09-20T02:28:45.725-07:00 WARN (REST) failed to bootstrap client, will retry: failed to execute request: Get "http://127.0.0.1:8091/pools/default/nodeServices": dial tcp 127.0.0.1:8091: connect: connection refused
      2022-09-20T02:28:45.725-07:00 ERROR (Main) Failed to run node {"err": "could not create REST client: failed to bootstrap client: failed to connect to any host(s) from the connection string, check username and password"}[ns_server:debug,2022-09-20T02:28:45.875-07:00,ns_1@s22401-ip6.qe.couchbase.com:cb_dist<0.211.0>:cb_dist:info_msg:872]cb_dist: Accepted new connection from <0.216.0> DistCtrl #Port<0.468>: {con,
                                                                              #Ref<0.1814526785.2013528065.208661>,
                                                                              inet6_tcp_dist,
                                                                              undefined,
                                                                              undefined}
      [

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Build couchbase-server-7.2.0-2142 contains backup commit 0a93674 with commit message:
          MB-53739 Update tools-common dependency

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-2142 contains backup commit 0a93674 with commit message: MB-53739 Update tools-common dependency

          Build couchbase-server-7.2.0-2142 contains cbbs commit a8f8d74 with commit message:
          MB-53739 Use localhost as the management API address

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-2142 contains cbbs commit a8f8d74 with commit message: MB-53739 Use localhost as the management API address

          Working fine on latest run using 7.1.2-3439.

          Closing this ticket

          ashwin.govindarajulu Ashwin Govindarajulu added a comment - Working fine on latest run using  7.1.2-3439. Closing this ticket

          Build couchbase-server-8.0.0-1122 contains backup commit 0a93674 with commit message:
          MB-53739 Update tools-common dependency

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1122 contains backup commit 0a93674 with commit message: MB-53739 Update tools-common dependency

          Build couchbase-server-8.0.0-1122 contains cbbs commit a8f8d74 with commit message:
          MB-53739 Use localhost as the management API address

          build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1122 contains cbbs commit a8f8d74 with commit message: MB-53739 Use localhost as the management API address

          People

            ashwin.govindarajulu Ashwin Govindarajulu
            ashwin.govindarajulu Ashwin Govindarajulu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty