Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46321

Investigate failure to start analytics service after cluster-init before timeout of 120s

    XMLWordPrintable

Details

    • Task
    • Resolution: Unresolved
    • Critical
    • backlog
    • Cheshire-Cat
    • ns_server
    • [ClusterExecutionIPv6LiteralIT 9: logging: get_loggers]
    • 1

    Description

      Filed as a task, as cbcollect_info logs are not available, but i have attached the contents of the logs dir as n_0_logs.zip, in hopes that it provides the needed info:

      2021-05-16T08:14:15.121-07:00 INFO ClusterExecutionITBase [main] Running cli: ip-family -c [::1]:9000 -u couchbase -p couchbase --get
      2021-05-16T08:14:15.352-07:00 INFO ClusterExecutionITBase [main+] >> Cluster using ipv4
      2021-05-16T08:14:15.378-07:00 INFO ClusterExecutionITBase [main] Running cli: node-to-node-encryption -c [::1]:9000 -u couchbase -p couchbase --get
      2021-05-16T08:14:15.605-07:00 INFO ClusterExecutionITBase [main+] >> Node-to-node encryption is disabled
      2021-05-16T08:14:15.624-07:00 INFO ClusterExecutionITBase [main] Running cli: ip-family -c [::1]:9000 -u couchbase -p couchbase --set --ipv6
      2021-05-16T08:14:19.397-07:00 INFO ClusterExecutionITBase [main+] >> SUCCESS: Switched IP family of the cluster
      2021-05-16T08:14:19.453-07:00 INFO ClusterExecutionITBase [pool-3] +http://[::1]:9000/node/controller/rename
      2021-05-16T08:14:19.671-07:00 INFO ClusterExecutionITBase [main] cluster is running
      2021-05-16T08:14:19.672-07:00 INFO ClusterExecutionITBase [pool-3] configuring cluster with services: [DATA, ANALYTICS]
      2021-05-16T08:14:19.672-07:00 INFO ClusterExecutionITBase [pool-3] Running cli: node-init -c [::1]:9000 -u couchbase -p couchbase --node-init-data-path /home/couchbase/jenkins/workspace/cbas-cbcluster-test2/ns_server/data/n_0/datadir --node-init-index-path /home/couchbase/jenkins/workspace/cbas-cbcluster-test2/ns_server/data/n_0/datadir --ipv6 --node-init-analytics-path /home/couchbase/jenkins/workspace/cbas-cbcluster-test2/ns_server/data/n_0/datadir --node-init-hostname ::1
      2021-05-16T08:14:19.893-07:00 INFO ClusterExecutionITBase [pool-3+] >> SUCCESS: Node initialized
      2021-05-16T08:14:19.912-07:00 INFO ClusterExecutionITBase [pool-3] Running cli: cluster-init --cluster-username couchbase --cluster-password couchbase -c [::1]:9000 --services data,analytics --cluster-analytics-ramsize 3072 --cluster-ramsize 3072
      2021-05-16T08:14:31.706-07:00 INFO ClusterExecutionITBase [pool-3+] >> SUCCESS: Cluster initialized
      2021-05-16T08:14:31.727-07:00 INFO ClusterExecutionITBase [main] state: [0], nodes: {{DATA,ANALYTICS}}, cbas node count: 1
      2021-05-16T08:14:31.727-07:00 INFO ClusterExecutionITBase [assertCbasNodesActive-10-[::1]:9600] Waiting for NCs == 1 & Cluster state in {ACTIVE} for up to 120s...
      2021-05-16T08:16:31.728-07:00 WARN ClusterExecutionITBase [assertCbasNodesActive-10-[::1]:9600] Timed out waiting for condition: NCs == 1 (last count was -1) & Cluster [ACTIVE] (last state was null), last error was org.apache.http.conn.HttpHostConnectException: Connect to [::1]:9600 [/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused), attemptCount: 120
      

      As far as I can tell from the logs, the cbas service is never started (no analytics logs are present), which is what causes this failure.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            michael.blow Michael Blow
            michael.blow Michael Blow
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty