Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.6.5
-
Centos 7 64 bit; CB 7.1.0-2140
-
Untriaged
-
1
-
Yes
-
Analytics Sprint 5
Description
The Analytics Service cannot be added nor re-added to a 6.6.5 cluster w/ node-to-node encryption configured at all or strict, attempts to do so will result in a failed rebalance as occurred in this issue. This is a regression only present in 6.6.5.
Steps to Repro
1. Create a 1 node KV cluster 172.23.122.100
2. Turn on n2n encryption and enforce TLS on the above node
3. Repeat the above two steps on 104 more nodes (which include 15 cbas nodes)
4. Add these 104 nodes to the master node (.100)
5. Rebalance the cluster. This initial rebalance fails as cbas keeps crashing
"completionMessage": "Rebalance exited with reason {service_rebalance_failed,cbas,\n {agent_died,<32913.19813.33>,\n {lost_connection,\n {'ns_1@172.23.122.128',shutdown}}}}."
|
},
|
Note
I retried the rebalance after ~ 20 minutes, but it still failed with the same issue.
UI logs are flooded with the following messages
Service 'cbas' exited with status 113. Restarting. Messages:
|
internal/poll.(*FD).Accept(0xc00058e600)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/internal/poll/fd_unix.go:402 +0x22c
|
net.(*netFD).accept(0xc00058e600)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/fd_unix.go:173 +0x35
|
net.(*TCPListener).accept(0xc00000ea98)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/tcpsock_posix.go:140 +0x28
|
net.(*TCPListener).Accept(0xc00000ea98)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/tcpsock.go:262 +0x3d
|
net/http.(*Server).Serve(0xc000302000, {0x962ed0, 0xc00000ea98})
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/http/server.go:3001 +0x394
|
net/http.(*Server).ListenAndServe(0xc000302000)
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.17.2/go/src/net/http/server.go:2930 +0x7d
|
main.StartHTTPServer.func1()
|
goproj/src/github.com/couchbase/cbas/cbas/httpserver.go:43 +0x25
|
created by main.StartHTTPServer
|
goproj/src/github.com/couchbase/cbas/cbas/httpserver.go:42 +0x28f
|
|
|
*** end; calling os.Exit()..
|
Attachments
Issue Links
- Clones
-
-
- Closed
-
- is a backport of
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-