Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53884

[BP 6.6.6] Analytics Service cannot be added nor re-added to a 6.6.5 cluster w/ node-to-node encryption configured

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Yes
    • Analytics Sprint 5

    Description

      The Analytics Service cannot be added nor re-added to a 6.6.5 cluster w/ node-to-node encryption configured at all or strict, attempts to do so will result in a failed rebalance as occurred in this issue. This is a regression only present in 6.6.5.

      Steps to Repro
      1. Create a 1 node KV cluster 172.23.122.100
      2. Turn on n2n encryption and enforce TLS on the above node
      3. Repeat the above two steps on 104 more nodes (which include 15 cbas nodes)
      4. Add these 104 nodes to the master node (.100)
      5. Rebalance the cluster. This initial rebalance fails as cbas keeps crashing

      "completionMessage": "Rebalance exited with reason {service_rebalance_failed,cbas,\n                              {agent_died,<32913.19813.33>,\n                               {lost_connection,\n                                {'ns_1@172.23.122.128',shutdown}}}}."
        },

      Note
      I retried the rebalance after ~ 20 minutes, but it still failed with the same issue.

      UI logs are flooded with the following messages

      Service 'cbas' exited with status 113. Restarting. Messages:
      internal/poll.(*FD).Accept(0xc00058e600)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/internal/poll/fd_unix.go:402 +0x22c
      net.(*netFD).accept(0xc00058e600)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/fd_unix.go:173 +0x35
      net.(*TCPListener).accept(0xc00000ea98)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/tcpsock_posix.go:140 +0x28
      net.(*TCPListener).Accept(0xc00000ea98)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/tcpsock.go:262 +0x3d
      net/http.(*Server).Serve(0xc000302000, {0x962ed0, 0xc00000ea98})
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/http/server.go:3001 +0x394
      net/http.(*Server).ListenAndServe(0xc000302000)
      /home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/http/server.go:2930 +0x7d
      main.StartHTTPServer.func1()
      goproj/src/github.com/couchbase/cbas/cbas/httpserver.go:43 +0x25
      created by main.StartHTTPServer
      goproj/src/github.com/couchbase/cbas/cbas/httpserver.go:42 +0x28f
       
      *** end; calling os.Exit()..

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-6.6.6-10515 contains cbas commit 99f1528 with commit message:
            MB-53884: [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.6-10515 contains cbas commit 99f1528 with commit message: MB-53884 : [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            Build couchbase-server-7.0.5-7626 contains cbas commit 99f1528 with commit message:
            MB-53884: [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.5-7626 contains cbas commit 99f1528 with commit message: MB-53884 : [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            Build couchbase-server-7.2.0-5055 contains cbas commit 99f1528 with commit message:
            MB-53884: [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.2.0-5055 contains cbas commit 99f1528 with commit message: MB-53884 : [BP 6.6.6] Do not kill/restart the Analytics driver if not running
            umang.agrawal Umang added a comment -

            Verified with Enterprise Edition 6.6.6 build 10570

            umang.agrawal Umang added a comment - Verified with Enterprise Edition 6.6.6 build 10570

            Build couchbase-server-8.0.0-1211 contains cbas commit 99f1528 with commit message:
            MB-53884: [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            build-team Couchbase Build Team added a comment - Build couchbase-server-8.0.0-1211 contains cbas commit 99f1528 with commit message: MB-53884 : [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            Build couchbase-server-7.5.0-3515 contains cbas commit 99f1528 with commit message:
            MB-53884: [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.5.0-3515 contains cbas commit 99f1528 with commit message: MB-53884 : [BP 6.6.6] Do not kill/restart the Analytics driver if not running

            People

              umang.agrawal Umang
              sumedh.basarkod Sumedh Basarkod (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty