* Multi-Dimensional Scaling
Service Total Nodes
---------------------------------
fts 2 172.23.99.{39,40}
kv 1 172.23.99.211Also...
[BAD] Linux OOM Killer : evidence of Linux OOM Killer in syslogs or dmesg in couchbase.log
!/Users/steveyen/nutshell/bin/nutshell cbcollect*
Unable to import Couchbase dependencies Note: Nutshell running under standard interpreter. Install PyPy for best performance. INFO|nutshell.parsers: Parsing logs for ['cbcollect_info_ns_1@172.23.99.211_20180531-210617', 'cbcollect_info_ns_1@172.23.99.39_20180531-210618', 'cbcollect_info_ns_1@172.23.99.40_20180531-210618'] now WARNING|nutshell.parsers:nutshell.py:323: Unable to open cbcollect_info_ns_1@172.23.99.211_20180531-210617/ns_server.xdcr.log WARNING|nutshell.parsers:nutshell.py:323: Unable to open cbcollect_info_ns_1@172.23.99.39_20180531-210618/ns_server.xdcr.log WARNING|nutshell.parsers:nutshell.py:323: Unable to open cbcollect_info_ns_1@172.23.99.40_20180531-210618/ns_server.xdcr.log WARNING|nutshell.parsers:nutshell.py:323: Unable to open cbcollect_info_ns_1@172.23.99.39_20180531-210618/ddocs.log WARNING|nutshell.parsers:nutshell.py:323: Unable to open cbcollect_info_ns_1@172.23.99.40_20180531-210618/ddocs.log INFO|cbcollect_info_ns_1@172.23.99.211_20180531-210617|nutshell.parsers: Parsing cbcollect_info_ns_1@172.23.99.211_20180531-210617/couchbase.log now INFO|cbcollect_info_ns_1@172.23.99.40_20180531-210618|nutshell.parsers: Parsing cbcollect_info_ns_1@172.23.99.40_20180531-210618/couchbase.log now INFO|cbcollect_info_ns_1@172.23.99.39_20180531-210618|nutshell.parsers: Parsing cbcollect_info_ns_1@172.23.99.39_20180531-210618/couchbase.log now WARNING|cbcollect_info_ns_1@172.23.99.39_20180531-210618|nutshell.parsers:cblogparser.py:785: Failed to parse ifconfig WARNING|cbcollect_info_ns_1@172.23.99.40_20180531-210618|nutshell.parsers:cblogparser.py:785: Failed to parse ifconfig WARNING|cbcollect_info_ns_1@172.23.99.211_20180531-210617|nutshell.parsers:cblogparser.py:785: Failed to parse ifconfig ERROR|cbcollect_info_ns_1@172.23.99.40_20180531-210618|nutshell.parsers:mcdstatsparser.py:83: Unexpected stats command in stats.log: Phosphor Trace INFO|cbcollect_info_ns_1@172.23.99.40_20180531-210618|nutshell.parsers: No ddocs.log Found INFO|cbcollect_info_ns_1@172.23.99.40_20180531-210618|nutshell.parsers: No ns_server.xdcr.log Found ERROR|cbcollect_info_ns_1@172.23.99.39_20180531-210618|nutshell.parsers:mcdstatsparser.py:83: Unexpected stats command in stats.log: Phosphor Trace INFO|cbcollect_info_ns_1@172.23.99.39_20180531-210618|nutshell.parsers: No ddocs.log Found INFO|cbcollect_info_ns_1@172.23.99.39_20180531-210618|nutshell.parsers: No ns_server.xdcr.log Found INFO|cbcollect_info_ns_1@172.23.99.40_20180531-210618|nutshell.parsers: Parse of cbcollect_info_ns_1@172.23.99.40_20180531-210618 complete. Took 26.669s INFO|cbcollect_info_ns_1@172.23.99.39_20180531-210618|nutshell.parsers: Parse of cbcollect_info_ns_1@172.23.99.39_20180531-210618 complete. Took 26.740s ERROR|cbcollect_info_ns_1@172.23.99.211_20180531-210617|nutshell.parsers:mcdstatsparser.py:83: Unexpected stats command in stats.log: Phosphor Trace INFO|cbcollect_info_ns_1@172.23.99.211_20180531-210617|nutshell.parsers: No ns_server.xdcr.log Found INFO|cbcollect_info_ns_1@172.23.99.211_20180531-210617|nutshell.parsers: Parse of cbcollect_info_ns_1@172.23.99.211_20180531-210617 complete. Took 30.438s INFO|nutshell.parsers: Finished parsing logs for ['cbcollect_info_ns_1@172.23.99.211_20180531-210617', 'cbcollect_info_ns_1@172.23.99.39_20180531-210618', 'cbcollect_info_ns_1@172.23.99.40_20180531-210618'] WARNING|nutshell.checkers:cluster.py:323: Unable to parse timestamp `2018-05-31T13:36:49.897-07:00` WARNING|nutshell.checkers:cluster.py:323: Unable to parse timestamp `2018-05-31T13:36:49.897-07:00` WARNING|nutshell.checkers:cluster.py:323: Unable to parse timestamp `2018-05-31T13:36:49.897-07:00` ===Checker results for 'fts01-s424' / 'ns_1@172.23.99.211'=== [info] Collection Time : 2018-05-31 21:09:06 [info] OS Name : Linux 3.10.0-514.26.2.el7.x86_64 [info] OS Version : CentOS Linux release 7.3.1611 (Core) [BAD] CB Version : 5.5.0-2780-enterprise - Unsupported build [info] CB Packages : couchbase-server-5.5.0-2780.x86_64 [info] CB Uptime : 32 minutes, 10 seconds [info] System Uptime : 95 days, 15:15:15 [ok] Installed CPUs : 48 [ok] Installed RAM : 257677 MB [ok] Used RAM : 37.1% (95685 / 257677 MB) [ok] Total Server Quota : Total Quota (51024 MB) 19.80% of Total RAM (257677 MB) (206653.0 MB) remaining for the OS [ok] Erlang VM vsize : 3,665 MB [ok] Memcached vsize : 34,745 MB [ok] Memcached Fragmentation: 0.21% (54 Fragmented MB / 25525 Total MB) [ok] vm.swappiness : 0 (Recommended value) [ok] Swap Used : 0.00% [info] Local Port Range : Not using default value. Range currently set at 32768 - 60999 but default value is 32768 - 61000 [ok] Transparent Huge Pages: never [warn] Slow operations : 1 - Last slow operation was at 2018-05-31T14:02:38.808449Z [info] Data Directory : (non-default) /data - ext4 - (/dev/sdc1 440GB / 5% used) [info] MDS Services : kv ===Analyser results=== * Connection Information Established TIME_WAIT Total TCP Connections in State Conns Hosts Conns Hosts Conns Hosts SendQ (Bytes) -------------------------------------------------------------------------------- Port 8091 (cluster mgmt) 8 1 0 0 9 2 0 Port 8092 (views) 0 0 0 0 1 1 0 Port 9999 (Projector) 0 0 0 0 1 1 0 Port 11207 (KV SSL) 0 0 0 0 2 2 0 Port 11209 (ERL -> memcached) 6 1 0 0 8 3 0 Port 11210 (KV) 4 1 0 0 6 3 0 Port 11211 (moxi) 0 0 0 0 0 0 0 Port 18091 (cluster mgmt SSL) 0 0 0 0 1 1 0 Port 18092 (views SSL) 0 0 0 0 1 1 0 ==Total== 54 5 1 1 77 7 47 Total recv queue (bytes): 0 * Open File Information Command FDs MMaps ----------------------- beam.smp 113 51 memcached 281 54 moxi 0 0 goxdcr 11 3 projector 8 17 ==Total== 422 129 * Per Node Bucket Stats (CB/Eph buckets only) Resident Ratio Bucket CurrItems KV (MB) Meta% Active Replica DWQ ------------------------------------------------------------------- bucket-1 10000000 25023 1.2 100 100 0 Total (1 buckets) 10000000 25023 - - - 0 * Clients Connected Client Port Times Connected Last Connection Unique IPs ------------------------------------------------------------------------------------------------------------------------------- cbdatasource-fts:perf_fts_index_51e1631267ac169a_13aa53f3-3f20ef65 11210 1 ( 0.32%) 2018-05-31T13:39:18.425 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_13aa53f3-6b50344d 11210 1 ( 0.32%) 2018-05-31T14:06:03.873 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_18572d87-6505a7b0 11210 1 ( 0.32%) 2018-05-31T13:39:18.136 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_54820232-7a3a141c 11210 1 ( 0.32%) 2018-05-31T14:06:03.874 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_54820232-b69e7b 11210 1 ( 0.32%) 2018-05-31T13:39:18.426 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_6ddbfb54-2a7fcda1 11210 1 ( 0.32%) 2018-05-31T13:39:18.138 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_aa574717-263f4b45 11210 1 ( 0.32%) 2018-05-31T14:06:03.873 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_aa574717-4428d136 11210 1 ( 0.32%) 2018-05-31T13:39:18.426 1 cbdatasource-fts:perf_fts_index_51e1631267ac169a_f4e0a48a-7572938d 11210 1 ( 0.32%) 2018-05-31T13:39:18.139 1 cbstats 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 25 ( 8.01%) 2018-05-31T14:02:45.855 1 couchbase-java-client/2.5.4 11210 84 ( 26.92%) 2018-05-31T13:49:03.511 1 gocbcore/v7.1.0 11210 48 ( 15.38%) 2018-05-31T13:37:11.235 1 mcbp 11209 73 ( 23.40%) 2018-05-31T14:03:07.219 1 mcctl 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 69 ( 22.12%) 2018-05-31T14:03:07.219 1 mcstat 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 2 ( 0.64%) 2018-05-31T14:02:45.982 1 mctimings 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 2 ( 0.64%) 2018-05-31T14:03:01.127 1 ===Checker results for '172-23-99-39' / 'ns_1@172.23.99.39'=== [info] Collection Time : 2018-05-31 21:08:24 [info] OS Name : Linux 3.10.0-514.26.2.el7.x86_64 [info] OS Version : CentOS Linux release 7.3.1611 (Core) [BAD] CB Version : 5.5.0-2780-enterprise - Unsupported build [info] CB Packages : couchbase-server-5.5.0-2780.x86_64 [warn] CB Uptime : ??? (Attribute not found). Cannot check. [info] System Uptime : 0:02:06 [ok] Installed CPUs : 48 [warn] Installed RAM : 7705 MB - less than recommended (16384 MB) [ok] Used RAM : 16.6% (1277 / 7705 MB) [ok] Total Server Quota : Total Quota (1000 MB) 12.98% of Total RAM (7705 MB) (less than maximum of 80.00%) [ok] Erlang VM vsize : 2,012 MB [ok] Memcached vsize : 1,625 MB [warn] Memcached Fragmentation: ??? (Attribute not found). Cannot check. [warn] vm.swappiness : 60 - Recommended value is 0 [ok] Swap Used : 0.00% [info] Local Port Range : Not using default value. Range currently set at 32768 - 60999 but default value is 32768 - 61000 [BAD] Transparent Huge Pages: always - Recommended is 'never' or 'madvise' [BAD] Linux OOM Killer : evidence of Linux OOM Killer in syslogs or dmesg in couchbase.log [warn] Slow operations : 14 - Last slow operation was at 2018-05-31T13:51:15.287210Z [info] MDS Services : fts [warn] NS Stats Ticks : 15 instances found, showing the last 3 : 2018-05-31T13:51:11.171-07:00 (Collector: global_stats_collector) Dropped 12 ticks : 2018-05-31T13:51:00.222-07:00 (Collector: global_stats_collector) Dropped 11 ticks : 2018-05-31T13:50:54.699-07:00 (Collector: service_stats_collector) Dropped 12 ticks ===Analyser results=== * Connection Information Established TIME_WAIT Total TCP Connections in State Conns Hosts Conns Hosts Conns Hosts SendQ (Bytes) -------------------------------------------------------------------------------- Port 8091 (cluster mgmt) 6 1 0 0 7 2 0 Port 8094 (FTS) 0 0 0 0 0 0 0 Port 18091 (cluster mgmt SSL) 0 0 0 0 1 1 0 Port 18094 (FTS SSL) 0 0 0 0 0 0 0 ==Total== 27 4 2 1 49 6 0 Total recv queue (bytes): 0 * Open File Information Command FDs MMaps ----------------------- beam.smp 97 50 cbft 0 0 ==Total== 388 111 * Clients Connected Client Port Times Connected Last Connection Unique IPs --------------------------------------------------------------------------------------------------------------- cbstats 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 25 ( 55.56%) 2018-05-31T14:01:46.535 1 mcbp 11209 10 ( 22.22%) 2018-05-31T14:01:49.414 1 mcctl 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 6 ( 13.33%) 2018-05-31T14:01:49.414 1 mcstat 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 2 ( 4.44%) 2018-05-31T14:01:46.620 1 mctimings 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 2 ( 4.44%) 2018-05-31T14:01:47.798 1 ===Checker results for '172-23-99-40' / 'ns_1@172.23.99.40'=== [info] Collection Time : 2018-05-31 21:08:38 [info] OS Name : Linux 3.10.0-514.26.2.el7.x86_64 [info] OS Version : CentOS Linux release 7.3.1611 (Core) [BAD] CB Version : 5.5.0-2780-enterprise - Unsupported build [info] CB Packages : couchbase-server-5.5.0-2780.x86_64 [warn] CB Uptime : ??? (Attribute not found). Cannot check. [info] System Uptime : 0:02:51 [ok] Installed CPUs : 48 [warn] Installed RAM : 7705 MB - less than recommended (16384 MB) [ok] Used RAM : 16.8% (1296 / 7705 MB) [ok] Total Server Quota : Total Quota (1000 MB) 12.98% of Total RAM (7705 MB) (less than maximum of 80.00%) [ok] Erlang VM vsize : 2,117 MB [ok] Memcached vsize : 2,265 MB [warn] Memcached Fragmentation: ??? (Attribute not found). Cannot check. [warn] vm.swappiness : 60 - Recommended value is 0 [ok] Swap Used : 0.00% [info] Local Port Range : Not using default value. Range currently set at 32768 - 60999 but default value is 32768 - 61000 [BAD] Transparent Huge Pages: always - Recommended is 'never' or 'madvise' [BAD] Linux OOM Killer : evidence of Linux OOM Killer in syslogs or dmesg in couchbase.log [info] MDS Services : fts ===Analyser results=== * Connection Information Established TIME_WAIT Total TCP Connections in State Conns Hosts Conns Hosts Conns Hosts SendQ (Bytes) -------------------------------------------------------------------------------- Port 8091 (cluster mgmt) 16 1 0 0 17 2 0 Port 8094 (FTS) 1 1 0 0 2 2 0 Port 18091 (cluster mgmt SSL) 0 0 0 0 1 1 0 Port 18094 (FTS SSL) 0 0 0 0 1 1 0 ==Total== 65 5 0 0 87 7 0 Total recv queue (bytes): 0 * Open File Information Command FDs MMaps ----------------------- beam.smp 112 50 cbft 104 82 ==Total== 514 193 * Clients Connected Client Port Times Connected Last Connection Unique IPs --------------------------------------------------------------------------------------------------------------- cbstats 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 25 ( 55.56%) 2018-05-31T14:01:53.960 1 mcbp 11209 10 ( 22.22%) 2018-05-31T14:01:56.281 1 mcctl 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 6 ( 13.33%) 2018-05-31T14:01:56.281 1 mcstat 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 2 ( 4.44%) 2018-05-31T14:01:54.082 1 mctimings 07ad8e796e55c364c44ce0950a0905049635fbc7 11209 2 ( 4.44%) 2018-05-31T14:01:54.976 1 ===Cluster '172.23.99.{39,40,211}' Information=== [info] UUID : 744641891cd147779ea31635b9c5a497 [info] Node Count : 3 [info] Auto-failover : 5 seconds [info] Auto-compaction : 30% (default) [info] External Authentication: Enabled [warn] Installed RAM different: Nodes `172.23.99.211`: 257677 : Nodes `172.23.99.{39,40}`: 7705 [warn] vm.swappiness different: Nodes `172.23.99.211`: 0 (Recommended value) : Nodes `172.23.99.{39,40}`: 60 - Recommended value is 0 [warn] Transparent Huge Pages different: Nodes `172.23.99.{39,40}`: always : Nodes `172.23.99.211`: never * Bucket Details Views/DDocs Bucket Type Quota (MB) Prio Evict Indx Reps Conf Cmpct Access Flush Prod Dev ------------------------------------------------------------------------------------------- bucket-1 CB 51024 Low Value Off Seq 30% SASL On 0/0 0/0 Total (1 buckets) - 51024 - - - - - - - 0/0 0/0 * vBucket Details ---Replicas--- Bucket Nodes Total Online 1 2 3 -------------------------------------------------- bucket-1 1 1024 1024 n/a n/a n/a [warn] `bucket-1` Replicas Mis-configured: Number of replicas (1) is not possible on 1 node * Multi-Dimensional Scaling Service Total Nodes --------------------------------- fts 2 172.23.99.{39,40} kv 1 172.23.99.211 * Rack Zone Awareness Information Server Group Nodes ------------------------------------ Group 1 172.23.99.{39,40,211}