===Checker results for '172.23.108.97'=== [info] Collection Time : 2017-05-31 06:08:46 [info] OS Name : Linux 2.6.32-358.el6.x86_64 [info] OS Version : CentOS release 6.4 (Final) [BAD] CB Version : 5.0.0-2958-enterprise - Unsupported build [info] CB Packages : couchbase-server-5.0.0-2958.x86_64 [info] CB Uptime : 1 day, 13 hours, 33 minutes, 21 seconds [info] System Uptime : 34 days [ok] Installed CPUs : 8 [ok] Installed RAM : 23367 MB [ok] Used RAM : 47.5% (11103 / 23367 MB) [info] Data Quota : Data Quota (9346 MB) 40.00% of Total RAM (23367 MB) [info] FTS Quota : FTS Quota (512 MB) 2.19% of Total RAM (23367 MB) [ok] Total Server Quota : Total Quota (9858 MB) 42.19% of Total RAM (23367 MB) (13509.0 MB) remaining for the OS [ok] Erlang VM vsize : 3,612 MB [ok] Memcached vsize : 9,271 MB [warn] Memcached Fragmentation: ??? (Attribute not found). Cannot check. [warn] vm.swappiness : 60 - Recommended value is 0 [warn] Swap Used : 3.99% - non-zero swap usage (431980 / 10829816 kB). [ok] Interface 'eth0' (172.23.108.97) failures: RX:0 / TX:0 [info] Interface 'eth1' Status: down [warn] SYN Flooding : Possible SYN flooding detected on port 8092 in dmesg/syslogs : Possible SYN flooding detected on port 8094 in dmesg/syslogs [ok] Transparent Huge Pages: No khugedpages proccess found. THP is disabled. [BAD] Linux OOM Killer : evidence of Linux OOM Killer in syslogs or dmesg in couchbase.log [warn] Slow operations : 34685 - Last slow operation was at 2017-05-30T22:53:16.105491-07:00 [info] Data Directory : (non-default) /data - ext4 - (/dev/mapper/VolGroup01-Data 247GB / 8% used) [info] Index Directory : (non-default) /data - ext4 - (/dev/mapper/VolGroup01-Data 247GB / 8% used) [warn] CB Storage Check : Data and Index on the same disk/s [warn] View Indexing 'default': Design Doc 'scale' took 1710.639 sec to update (average 5.25 sec) [warn] View Indexing 'ORDER_LINE': Design Doc 'all' took 443.129 sec to update (average 28.484 sec) [warn] MDS Services : fts, kv - Only run 1 service per node in production - FTS is not supported [warn] NS Stats Ticks : 5223 instances found, showing the last 3 : 2017-05-30T22:55:10.198-07:00 (Collector: stats_collector) Dropped 7 ticks : 2017-05-30T22:55:10.183-07:00 (Collector: stats_collector) Dropped 7 ticks : 2017-05-30T22:55:10.157-07:00 (Collector: stats_collector) Dropped 7 ticks [BAD] Process Max Wait : Max wait for mc:writer_1 exceeded 1000ms (1193ms detected) : Max wait for mc:nonIO_0 exceeded 1000ms (1286ms detected) : Max wait for mc:nonIO_1 exceeded 1000ms (1367ms detected) : Max wait for mc:nonIO_2 exceeded 1000ms (1300ms detected) : Max wait for beam.smp exceeded 1000ms (1666ms detected) [BAD] Rebalance data loss : Bucket `CUSTOMER` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:35.631-07 : Bucket `DISTRICT` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:39.506-07 : Bucket `ORDER_LINE` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:44.914-07 : Bucket `default` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:33.869-07 : Bucket `NEW_ORDER` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:41.082-07 : Bucket `ITEM` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:36.881-07 : Bucket `WAREHOUSE` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:40.414-07 : Bucket `HISTORY` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:42.651-07 : Bucket `ORDERS` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:43.642-07 : Bucket `STOCK` experienced 5 loss(es) on rebalance. Last: 9% of data at 2017-05-30T20:48:38.249-07 ===Analyser results=== * Connection Information Established TIME_WAIT Total TCP Connections in State Conns Hosts Conns Hosts Conns Hosts SendQ (Bytes) -------------------------------------------------------------------------------- Port 8091 (cluster mgmt) 38 2 1 1 40 3 0 Port 8092 (views) 56 2 0 0 57 3 3698 Port 8094 (FTS) 1129 1 1690 1 9473 1 0 Port 9999 (Projector) 1 1 0 0 2 1 0 Port 11207 (KV SSL) 0 0 0 0 2 2 0 Port 11209 (ERL -> memcached) 120 5 0 0 122 7 0 Port 11210 (KV) 108 4 0 0 110 6 0 Port 11211 (moxi) 0 0 0 0 2 2 0 Port 18091 (cluster mgmt SSL) 0 0 0 0 1 1 0 Port 18092 (views SSL) 0 0 0 0 1 1 0 Port 18094 (FTS SSL) 0 0 0 0 1 1 0 ==Total== 1658 11 1903 9 10240 12 10351 Total recv queue (bytes): 516949 * Open File Information Command FDs MMaps ------------------------ beam.smp 355 64 cbft 0 0 memcached 908 49 moxi 43 30 goxdcr 35 3 projector 12 17 ==Total== 1362 167 * Per Node Bucket Stats (CB buckets only) Resident Ratio Bucket CurrItems KV (MB) Meta% Active Replica DWQ --------------------------------------------------------------------- CUSTOMER 34942 60 0.9 100 100 46 default 1795936 3750 1.3 40 6 3166 DISTRICT 12 0 0.0 100 100 0 HISTORY 8102 3 0.5 100 100 0 ITEM 21741 8 1.3 100 100 0 NEW_ORDER 1195 0 0.0 100 100 0 ORDER_LINE 324566 176 4.4 100 100 0 ORDERS 32171 12 0.4 100 100 0 STOCK 114534 131 2.6 100 100 0 WAREHOUSE 1 0 0.0 100 100 0 Total (10 buckets) 2333200 4140 - - - 3212 [warn] Resident Items 'default': % Resident Ratio is too low ===Cluster '172.23.{98.135,108.96,108.97,108.100,108.103,108.104}' Information=== [info] UUID : 35499887688996cf8a0b0e8132c0e424 [info] Node Count : 6 [info] Orchestrator : 172.23.108.103 [info] Auto-failover : Disabled (was 120 seconds) [info] Auto-compaction : 30% (default) * Bucket Details Views/DDocs Bucket Type Quota (MB) Prio Evict Indx Reps Conf Cmpct Access Flush Prod Dev -------------------------------------------------------------------------------------------- CUSTOMER CB 467 Low Value On Seq 30% SASL On 0/0 0/0 default CB 4673 Low Full On Seq 30% SASL On 3/1 0/0 DISTRICT CB 186 Low Value On Seq 30% SASL On 0/0 0/0 HISTORY CB 186 Low Value On Seq 30% SASL On 0/0 0/0 ITEM CB 186 Low Value On Seq 30% SASL On 0/0 0/0 NEW_ORDER CB 467 Low Value On Seq 30% SASL On 0/0 0/0 ORDER_LINE CB 934 Low Value On Seq 30% SASL On 1/1 0/0 ORDERS CB 934 Low Value On Seq 30% SASL On 0/0 0/0 STOCK CB 467 Low Value On Seq 30% SASL On 0/0 0/0 WAREHOUSE CB 467 Low Value On Seq 30% SASL On 0/0 0/0 Total (10 buckets) - 8967 - - - - - - - 4/2 0/0 * vBucket Details ---Replicas--- Bucket Nodes Total Online 1 2 3 ---------------------------------------------------- CUSTOMER 4 1024 1024 1024 n/a n/a default 4 1024 1024 860 n/a n/a DISTRICT 4 1024 1024 1024 n/a n/a HISTORY 4 1024 1024 1024 n/a n/a ITEM 4 1024 1024 1024 n/a n/a NEW_ORDER 4 1024 1024 1024 n/a n/a ORDER_LINE 4 1024 1024 1024 n/a n/a ORDERS 4 1024 1024 1024 n/a n/a STOCK 4 1024 1024 1024 n/a n/a WAREHOUSE 4 1024 1024 1024 n/a n/a [BAD] Replica-1 vbuckets for 'default': 164 are offline * Multi-Dimensional Scaling Service Total Nodes ------------------------------------------------------ fts 1 172.23.108.97 index 1 172.23.108.96 kv 4 172.23.{98.135,108.97,108.103,108.104} n1ql 1 172.23.108.100 * XDCR Replications Local Bucket Remote Cluster Remote Bucket Type ---------------------------------------------------------------------- default c4dd06bdced4386643f0e3ea5c086c8f remote xmem * Rack Zone Awareness Information Server Group Nodes -------------------------------------------------------------------- Group 1 172.23.{98.135,108.96,108.97,108.100,108.103,108.104} Source Code