Major Description
Problem
I have seen this problem a few times, when IPv6 is configured. When Couchbase Server is restarted, ns_server becomes unresponsive and I can't log in via the UI
From top you can see the CPU usage of beam jumping from ~75% to ~95%
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
|
4555 couchba+ 20 0 2145352 799888 0 S 91.6 78.6 1:42.88 beam.smp
|
From the debug.log it looks like the test bucket is not ready but also it looks like it has stopped logging too:
[root@node1-cb510-centos7 logs]# date
|
Fri 2 Mar 14:58:00 UTC 2018
|
[root@node1-cb510-centos7 logs]# tail -f debug.log
|
[ns_server:warn,2018-03-02T14:56:09.575Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:10.576Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:11.579Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:12.580Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:13.582Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:info,2018-03-02T14:56:14.541Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:<0.554.0>:ns_orchestrator:handle_info:472]Skipping janitor in state janitor_running
|
[ns_server:warn,2018-03-02T14:56:14.583Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:15.584Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:16.585Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
[ns_server:warn,2018-03-02T14:56:17.586Z,ns_1@fd63:6f75:6368:20d4:6066:55e0:e60f:9c9e:kv_monitor<0.580.0>:kv_monitor:get_buckets:275]The following buckets are not ready: ["test"]
|
I tried to restart via the service command but that hanged, did a kill then started it again but it ended up in the same state.
The only way I have worked out how to resolve the problem is to do a reinstall.
Setup
- One node cluster using Vagrants
- build: 5.5.0-1979
- Enabled IPv6 via the static config
- 1 bucket called test with 10,000 items