Details
Minor Description
While verifying the fix for - https://issues.couchbase.com/browse/MB-30093
Noticed below exceptions on UI after CC swap rebalance. Rebalance and running analytics query is not impacted despite seeing the below exceptions(Only on Vagrant not seen on VM's).
Service 'cbas' exited with status 1. Restarting. Messages: 2018-06-13T23:46:01.549+00:00 INFO CBAS.cbas Notified path /cbas/config/node/6ba6287409ccb73bb4095da2e77bc7e4 changed to {"configVersion":1,"config":{"additionalCcNodes":[],"address":"0.0.0.0","analyticsCcHttpPort":"9111","analyticsHttpAdminListenPort":"9110","analyticsHttpListenPort":"8095","cbasPort":"9122","ccAddress":"0.0.0.0","ccAnalyticsHttpListenPort":"9111","ccAnalyticsMetadataPartitionId":1,"ccClientListenPort":"9113","ccClusterListenPort":"9112","ccConsoleListenPort":"9114","ccControllerId":"2","ccHeartbeatMaxMisses":"4","ccHeartbeatPeriod":"30000","ccPublicAddress":"10.111.181.104","clusterAddress":"10.111.181.104","clusterListenPort":"9115","clusterPort":"9112","dataListenPort":"9116","defaultDir":"/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0","initialRun":false,"iodevices":["/opt/couchbase/var/lib/couchbase/data/@analytics/v_iodevice_0"],"logDir":"/opt/couchbase/var/lib/couchbase/logs","logLevel":"INFO","messagingListenPort":"9118","metadataCallbackPort":"9119","metadataListenPort":"9121","metadataNode":"6ba6287409ccb73bb4095da2e77bc7e4","ncservicePort":"-1","nsServerAddress":"127.0.0.1","nsServerPort":8091,"publicAddress":"10.111.181.104","replicationEnabled":true,"replicationFactor":2,"replicationListenPort":9120,"replicationStrategy":"metadata","resultListenPort":"9117","startingPartitionId":"2"},"nodeId":"6ba6287409ccb73bb4095da2e77bc7e4"} 2018-06-13T23:46:01.577+00:00 FATA CBAS.cbas error cbauth.SetRequestAuth(): Unable to find given hostport in cbauth database: `10.111.181.103:9110' [goport(/opt/couchbase/bin/cbas)] 2018/06/13 23:46:01 child process exited with status 1 |
Also, after a few minutes see indexer crashed
Service 'indexer' exited with status 134. Restarting. Messages: |
runtime.gopark(0x106af60, 0x0, 0xf8a15a, 0x6, 0x18, 0x2) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/proc.go:259 +0x13a fp=0xc425707458 sp=0xc425707428 |
runtime.selectgoImpl(0xc425707718, 0x0, 0x18) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/select.go:423 +0x1235 fp=0xc4257076b8 sp=0xc425707458 |
runtime.selectgo(0xc425707718) |
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/select.go:238 +0x1c fp=0xc4257076e0 sp=0xc4257076b8 |
github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0xc425c56350, 0xc4232f4fc0) |
godeps/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x13f fp=0xc4257077a8 sp=0xc4257076e0 |
runtime.goexit()
|
/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4257077b0 sp=0xc4257077a8 |
created by github.com/couchbase/cbauth/service.withTimeout
|
godeps/src/github.com/couchbase/cbauth/service/revrpc.go:43 +0xf6 |
[goport(/opt/couchbase/bin/indexer)] 2018/06/13 23:54:43 child process exited with status 134 |
steps:
1. Cluster with 1 KV and 2 CBAS node
2. Load beer-sample on KV and create a dataset on CBAS
3. Swap rebalance CC node and execute a count query on Analytics
4. We are seeing exceptions related to 'cbas' and 'indexer'