./testrunner -i /tmp/tuqvm.ini -t 2i.indexscans_2i.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index,groups=simple:equals:no_orderby_groupby:range,dataset=default,use_gsi_for_primary=True,use_gsi_for_secondary=True,standard_buckets=1,sasl_buckets=1,doc-per-day=50,nodes_init=4,verify_using_index_status=True,get-cbcollect-info=True,get-logs=True,get-coredumps=True Test Input params: {'get-coredumps': 'True', 'doc-per-day': '50', 'use_gsi_for_primary': 'True', 'use_gsi_for_secondary': 'True', 'standard_buckets': '1', 'dataset': 'default', 'cluster_name': 'tuqvm', 'verify_using_index_status': 'True', 'ini': '/tmp/tuqvm.ini', 'groups': 'simple:equals:no_orderby_groupby:range', 'case_number': 1, 'get-logs': 'True', 'sasl_buckets': '1', 'get-cbcollect-info': 'True', 'logs_folder': '/data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1', 'num_nodes': 6, 'nodes_init': '4'} Run before suite setup for 2i.indexscans_2i.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'4.0.0-3232-enterprise'] 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_multi_create_query_explain_drop_index============== 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.107 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.108 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.110 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.111 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.112 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.109 2015-06-21 22:57:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.107:8091 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.107:8091 is running 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.108:8091 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.108:8091 is running 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.110:8091 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.110:8091 is running 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.111:8091 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.111:8091 is running 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.112:8091 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.112:8091 is running 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.109:8091 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.109:8091 is running 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ============== 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.107 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.108 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.110 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.111 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.112 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.109 2015-06-21 22:57:13 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.107:8091 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.107:8091 is running 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.108:8091 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.108:8091 is running 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.110:8091 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.110:8091 is running 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.111:8091 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.111:8091 is running 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.112:8091 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.112:8091 is running 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.109:8091 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.109:8091 is running 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ============== Cluster instance shutdown with force 2015-06-21 22:57:23 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2015-06-21 22:57:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.106.107: 8091:services=n1ql%2Ckv&password=password&hostname=172.23.106.107&user=Administrator 2015-06-21 22:57:24 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] http://172.23.106.107:8091//node/controller/setupServices error 400 reason: status: 400, content: cannot change node services after cluster is provisioned cannot change node services after cluster is provisioned 2015-06-21 22:57:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2015-06-21 22:57:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.107:8091:username=Administrator&password=password&port=8091 2015-06-21 22:57:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2031 2015-06-21 22:57:24 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.108:8091:username=Administrator&password=password&port=8091 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2031 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.110:8091:username=Administrator&password=password&port=8091 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2031 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.111:8091:username=Administrator&password=password&port=8091 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2031 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.112:8091:username=Administrator&password=password&port=8091 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2031 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.106.109:8091:username=Administrator&password=password&port=8091 2015-06-21 22:57:25 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2031 2015-06-21 22:57:25 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2015-06-21 22:57:26 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.108:8091 to cluster 2015-06-21 22:57:26 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.108:8091 to this cluster @172.23.106.107:8091 2015-06-21 22:57:31 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.110:8091 to cluster 2015-06-21 22:57:31 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.110:8091 to this cluster @172.23.106.107:8091 2015-06-21 22:57:37 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.106.111:8091 to cluster 2015-06-21 22:57:37 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.106.111:8091 to this cluster @172.23.106.107:8091 2015-06-21 22:57:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.106.108%2Cns_1%40172.23.106.107%2Cns_1%40172.23.106.111%2Cns_1%40172.23.106.110 2015-06-21 22:57:42 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2015-06-21 22:57:42 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2015-06-21 22:57:52 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.029282093 sec 2015-06-21 22:57:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.107:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=677&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None 2015-06-21 22:57:53 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2015-06-21 22:57:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.106.107 to accept set ops 2015-06-21 22:57:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 default 2015-06-21 22:57:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 default 2015-06-21 22:57:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 default 2015-06-21 22:57:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 default 2015-06-21 22:57:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 default 2015-06-21 22:57:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 default 2015-06-21 22:57:56 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'default' was created with per node RAM quota: 677 2015-06-21 22:57:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.107:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=677&proxyPort=11211&authType=sasl&name=bucket0&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=password 2015-06-21 22:57:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket bucket0 2015-06-21 22:57:57 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : bucket0 in 172.23.106.107 to accept set ops 2015-06-21 22:57:59 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 bucket0 2015-06-21 22:57:59 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 bucket0 2015-06-21 22:57:59 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 bucket0 2015-06-21 22:58:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 bucket0 2015-06-21 22:58:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 bucket0 2015-06-21 22:58:00 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 bucket0 2015-06-21 22:58:01 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'bucket0' was created with per node RAM quota: 677 2015-06-21 22:58:02 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.106.107:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket0&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=677 2015-06-21 22:58:02 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.02 seconds to create bucket standard_bucket0 2015-06-21 22:58:02 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : standard_bucket0 in 172.23.106.107 to accept set ops 2015-06-21 22:58:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 standard_bucket0 2015-06-21 22:58:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 standard_bucket0 2015-06-21 22:58:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 standard_bucket0 2015-06-21 22:58:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 standard_bucket0 2015-06-21 22:58:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 standard_bucket0 2015-06-21 22:58:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 standard_bucket0 2015-06-21 22:58:05 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'standard_bucket0' was created with per node RAM quota: 677 2015-06-21 22:58:05 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_multi_create_query_explain_drop_index ============== 2015-06-21 22:58:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.107 cluster is warmup 2015-06-21 22:58:05 | INFO | MainProcess | test_thread | [newtuq.setUp] Waiting for cluster to become healthy 2015-06-21 22:58:05 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.107 is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.108 cluster is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.108 is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.110 cluster is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.110 is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.111 cluster is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.111 is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.112 cluster is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.112 is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] Initial status of 172.23.106.109 cluster is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [newtuq.setUp] current status of 172.23.106.109 is healthy 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.106.107 with username : root password : couchbase ssh_key: 2015-06-21 22:58:10 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.106.107 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [basetestcase.load] create 100800 to bucket0 documents... 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 bucket0 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 bucket0 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 bucket0 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [basetestcase.load] create 100800 to default documents... 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 default 2015-06-21 22:58:34 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 default 2015-06-21 22:58:35 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 default 2015-06-21 22:58:35 | INFO | MainProcess | test_thread | [basetestcase.load] create 100800 to standard_bucket0 documents... 2015-06-21 22:58:35 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.107:11210 standard_bucket0 2015-06-21 22:58:35 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.108:11210 standard_bucket0 2015-06-21 22:58:36 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.106.110:11210 standard_bucket0 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.107 port:8091 ssh_username:root] 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [newtuq.setUp] ip:172.23.106.107 port:8091 ssh_username:root 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] CREATE PRIMARY INDEX ON bucket0 USING GSI 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 634.361586ms 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON bucket0 USING GSI 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+bucket0++USING+GSI 2015-06-21 23:04:35 | ERROR | MainProcess | test_thread | [tuq_helper.create_primary_index] ERROR during index creation CBQError: host 172.23.106.107: ERROR:{u'status': u'stopped', u'errors': [{u'msg': u'Authorization Failed Keyspace bucket0', u'code': 10000}], u'results': [], u'metrics': {u'elapsedTime': u'6.227579ms', u'executionTime': u'6.059373ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'0aa3806b-6496-4b37-a523-43f87d4b5233', u'signature': None} 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] CREATE PRIMARY INDEX ON default USING GSI 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:04:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:04:36 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 292.292863ms 2015-06-21 23:04:36 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON default USING GSI 2015-06-21 23:04:36 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+default++USING+GSI 2015-06-21 23:04:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 10.644440424s 2015-06-21 23:04:46 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:04:46 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 829.930432ms 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [tuq_helper.create_primary_index] CREATE PRIMARY INDEX ON standard_bucket0 USING GSI 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 274.011673ms 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE PRIMARY INDEX ON standard_bucket0 USING GSI 2015-06-21 23:04:47 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=CREATE+PRIMARY+INDEX+ON+standard_bucket0++USING+GSI 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 10.148723602s 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 441.442892ms 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [base_2i.setUp] {'in_between': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'initial': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'after': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}, 'before': {'query_explain_ops': False, 'create_index': False, 'drop_index': False, 'query_ops': False}} 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.107 port:8091 ssh_username:root] 2015-06-21 23:04:58 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.111 port:8091 ssh_username:root] 2015-06-21 23:04:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee5d986f69010b4ab88a051dc199f414f7job_title ON bucket0(job_title) USING GSI WITH {'nodes': ['172.23.106.111:8091']} 2015-06-21 23:04:59 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee5d986f69010b4ab88a051dc199f414f7job_title+ON+bucket0%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.106.111%3A8091%27%5D%7D 2015-06-21 23:04:59 | ERROR | MainProcess | Cluster_Thread | [task.execute] CBQError: host 172.23.106.107: ERROR:{u'status': u'stopped', u'errors': [{u'msg': u'Authorization Failed Keyspace bucket0', u'code': 10000}], u'results': [], u'metrics': {u'elapsedTime': u'12.493055ms', u'executionTime': u'12.325664ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'f610a59e-b401-42b2-a180-a62b656c5569', u'signature': None} [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 71, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 2137, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Sun Jun 21 23:04:59 2015 2015-06-21 23:04:59 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee5d986f69010b4ab88a051dc199f414f7job_title ON default(job_title) USING GSI WITH {'nodes': ['172.23.106.111:8091']} [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/2i/indexscans_2i.py', 45, 'test_multi_create_query_explain_drop_index', 'self._create_index_in_async()'), ('pytests/2i/base_2i.py', 685, '_create_index_in_async', 'task.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')] 2015-06-21 23:04:59 | INFO | MainProcess | test_thread | [indexscans_2i.test_multi_create_query_explain_drop_index] CBQError: host 172.23.106.107: ERROR:{u'status': u'stopped', u'errors': [{u'msg': u'Authorization Failed Keyspace bucket0', u'code': 10000}], u'results': [], u'metrics': {u'elapsedTime': u'12.493055ms', u'executionTime': u'12.325664ms', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'f610a59e-b401-42b2-a180-a62b656c5569', u'signature': None} 2015-06-21 23:04:59 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee5d986f69010b4ab88a051dc199f414f7job_title+ON+default%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.106.111%3A8091%27%5D%7D 2015-06-21 23:05:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 16.846573247s 2015-06-21 23:05:16 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY CREATE INDEX employee5d986f69010b4ab88a051dc199f414f7job_title ON standard_bucket0(job_title) USING GSI WITH {'nodes': ['172.23.106.111:8091']} 2015-06-21 23:05:16 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=CREATE+INDEX+employee5d986f69010b4ab88a051dc199f414f7job_title+ON+standard_bucket0%28job_title%29+USING+GSI++WITH+%7B%27nodes%27%3A+%5B%27172.23.106.111%3A8091%27%5D%7D 2015-06-21 23:05:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 16.818026913s 2015-06-21 23:05:32 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:33 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 694.726333ms 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 395.161335ms 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 290.603965ms 2015-06-21 23:05:34 | ERROR | MainProcess | Cluster_Thread | [task.execute] Unexpected Exception Caught [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 71, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 2284, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Sun Jun 21 23:05:34 2015 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes ERROR [('/usr/lib64/python2.7/threading.py', 784, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 811, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/2i/indexscans_2i.py', 59, 'test_multi_create_query_explain_drop_index', 'self._run_tasks(tasks)'), ('pytests/2i/indexscans_2i.py', 342, '_run_tasks', 'task.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')] 2015-06-21 23:05:34 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:34 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of all nodes in cluster: [ip:172.23.106.107 port:8091 ssh_username:root] 2015-06-21 23:05:34 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] CHECK FOR PRIMARY INDEXES 2015-06-21 23:05:34 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON bucket0 USING GSI 2015-06-21 23:05:34 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:34 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 293.194707ms 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX default.employee5d986f69010b4ab88a051dc199f414f7job_title USING GSI 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=DROP+INDEX+default.employee5d986f69010b4ab88a051dc199f414f7job_title+USING+GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 332.672001ms 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON default USING GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 60.014585ms 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 302.681394ms 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON default USING GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+default+USING+GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 58.097593ms 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.drop_primary_index] DROP PRIMARY INDEX ON standard_bucket0 USING GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 365.381196ms 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY DROP INDEX standard_bucket0.employee5d986f69010b4ab88a051dc199f414f7job_title USING GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=DROP+INDEX+standard_bucket0.employee5d986f69010b4ab88a051dc199f414f7job_title+USING+GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 413.908062ms 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] RUN QUERY DROP PRIMARY INDEX ON standard_bucket0 USING GSI 2015-06-21 23:05:35 | INFO | MainProcess | test_thread | [rest_client.query_tool] query params : statement=DROP+PRIMARY+INDEX+ON+standard_bucket0+USING+GSI 2015-06-21 23:05:35 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 392.648555ms 2015-06-21 23:05:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:36 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 218.563166ms 2015-06-21 23:05:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 293.700987ms 2015-06-21 23:05:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] RUN QUERY SELECT * FROM system:indexes 2015-06-21 23:05:36 | INFO | MainProcess | Cluster_Thread | [rest_client.query_tool] query params : statement=SELECT+%2A+FROM+system%3Aindexes 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: cat /proc/cpuinfo 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: df -Th 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: cat /proc/meminfo 2015-06-21 23:05:36 | INFO | MainProcess | Cluster_Thread | [tuq_helper.run_cbq_query] TOTAL ELAPSED TIME: 290.989012ms 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: hostname 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: hostname -d 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: rm -f /opt/couchbase/var/lib/couchbase/meta;rm -f /tmp/log_upr_client.sock 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: killall cbq-engine 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:36 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: killall tuqtng 2015-06-21 23:05:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: killall indexer 2015-06-21 23:05:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:05:37 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_multi_create_query_explain_drop_index ============== 2015-06-21 23:05:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [u'bucket0', u'default', u'standard_bucket0'] on 172.23.106.107 2015-06-21 23:05:37 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket bucket0 ... 2015-06-21 23:05:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : bucket0 from 172.23.106.107 2015-06-21 23:05:39 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2015-06-21 23:05:39 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.107 existing buckets : [u'default', u'standard_bucket0'] 2015-06-21 23:05:39 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket default ... 2015-06-21 23:05:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : default from 172.23.106.107 2015-06-21 23:05:41 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2015-06-21 23:05:41 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.107 existing buckets : [u'standard_bucket0'] 2015-06-21 23:05:41 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] remove bucket standard_bucket0 ... 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleted bucket : standard_bucket0 from 172.23.106.107 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.wait_for_bucket_deletion] waiting for bucket deletion to complete.... 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [rest_client.bucket_exists] node 172.23.106.107 existing buckets : [] 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.108 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.110 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.111 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.112 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 172.23.106.109 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] rebalancing all nodes in order to remove nodes 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance params : password=password&ejectedNodes=ns_1%40172.23.106.108%2Cns_1%40172.23.106.111%2Cns_1%40172.23.106.110&user=Administrator&knownNodes=ns_1%40172.23.106.108%2Cns_1%40172.23.106.107%2Cns_1%40172.23.106.111%2Cns_1%40172.23.106.110 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [rest_client.rebalance] rebalance operation started 2015-06-21 23:05:43 | INFO | MainProcess | test_thread | [rest_client._rebalance_progress] rebalance percentage : 0.00 % 2015-06-21 23:05:53 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] rebalance progress took 10.0457649231 seconds 2015-06-21 23:05:53 | INFO | MainProcess | test_thread | [rest_client.monitorRebalance] sleep for 10 seconds after rebalance... 2015-06-21 23:06:03 | INFO | MainProcess | test_thread | [cluster_helper.cleanup_cluster] removed all the nodes from cluster associated with ip:172.23.106.107 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.108', 8091), (u'ns_1@172.23.106.111', 8091), (u'ns_1@172.23.106.110', 8091)] 2015-06-21 23:06:03 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.107:8091 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.107:8091 is running 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.108:8091 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.108:8091 is running 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.110:8091 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.110:8091 is running 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.111:8091 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.111:8091 is running 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.112:8091 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.112:8091 is running 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.106.109:8091 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.106.109:8091 is running 2015-06-21 23:06:13 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_multi_create_query_explain_drop_index ============== ====================================================================== ERROR: test_multi_create_query_explain_drop_index (2i.indexscans_2i.SecondaryIndexingScanTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/2i/indexscans_2i.py", line 59, in test_multi_create_query_explain_drop_index self._run_tasks(tasks) File "pytests/2i/indexscans_2i.py", line 342, in _run_tasks task.result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception DropIndexException: unable to drop index index employee5d986f69010b4ab88a051dc199f414f7job_title does not exist will not drop ---------------------------------------------------------------------- Ran 1 test in 550.798s FAILED (errors=1) downloading 172.23.106.107 ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.108 .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.110 ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.111 ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.112 ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.109 ......................................................................................................................................................................................................................................Cluster instance shutdown with force grabbing diags from http://172.23.106.107:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1/172.23.106.107-diag.txt.gz grabbing diags from http://172.23.106.108:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1/172.23.106.108-diag.txt.gz grabbing diags from http://172.23.106.110:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1/172.23.106.110-diag.txt.gz grabbing diags from http://172.23.106.111:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1/172.23.106.111-diag.txt.gz grabbing diags from http://172.23.106.112:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1/172.23.106.112-diag.txt.gz grabbing diags from http://172.23.106.109:8091/diag downloaded and zipped diags @ : /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1/172.23.106.109-diag.txt.gz grabbing cbcollect from 172.23.106.107 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.107 with username : root password : couchbase ssh_key: 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.107 Collecting logs from 172.23.106.107 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: cat /proc/cpuinfo 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: df -Th 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: cat /proc/meminfo 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: hostname 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: hostname -d 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:10:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: /opt/couchbase/bin/cbcollect_info 172.23.106.107-20150621-2310-diag.zip 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] uname (uname -a) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] time and TZ (date; date -u) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/timezone (cat /etc/timezone) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System Hardware (lshw -json || lshw) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/vmstat (cat /proc/vmstat) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/mounts (cat /proc/mounts) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/partitions (cat /proc/partitions) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/diskstats (cat /proc/diskstats) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/interrupts (cat /proc/interrupts) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (free -t) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (swapon -s) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel modules (lsmod) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/redhat-release) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (lsb_release -a) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/SuSE-release) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/issue) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (rpm -qa) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (COLUMNS=300 dpkg -l) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] sysctl settings (sysctl -a) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (lvdisplay) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (vgdisplay) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (pvdisplay) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (ifconfig -a) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Taking sample 2 after 10.000000 seconds - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/net/dev (cat /proc/net/dev) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network link statistics (ip -s link) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network status (netstat -anp || netstat -an) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network routing table (netstat -rn) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network socket statistics (ss -an) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Arp cache (arp -na) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Iptables dump (iptables-save) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/hosts (cat /etc/hosts) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Filesystem (df -ha) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System activity reporter (sar 1 10) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System paging activity (vmstat 1 10) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System uptime (uptime) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user definition (getent passwd couchbase) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Interrupt status (intrstat 1 10) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processor status (mpstat 1 10) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System log (cat /var/adm/messages) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/uptime (cat /proc/uptime) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --hardware) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --show) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (cat /sys/devices/system/node/node*/numastat) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel log buffer (dmesg) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network statistics (netstat -s) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Full raw netstat (cat /proc/net/netstat) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] product diag header () - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Version file (cat '/opt/couchbase/VERSION.txt') - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.txt') - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.xml') - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase config () - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase static config () - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw ns_log () - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (info.log) (cbbrowse_logs info.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (error.log) (cbbrowse_logs error.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (views.log) (cbbrowse_logs views.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (query.log) (cbbrowse_logs query.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '89c1aede6ae100e6f7fc97084970baad', 'connections']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] mctimings (['mctimings', '-u', '_admin', '-P', '89c1aede6ae100e6f7fc97084970baad', '-h', '127.0.0.1:11210', '-v']) - OK 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/couchbase.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/couchbase.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ddocs.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ddocs.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.error.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.ns_couchdb.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/stats.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/stats.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/memcached.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/memcached.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.ssl_proxy.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.xdcr_errors.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.http_access_internal.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/syslog.tar.gz 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.reports.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.query.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.couchdb.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.goxdcr.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.indexer.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.info.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/diag.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/diag.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.stats.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.xdcr.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.views.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ini.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ini.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.mapreduce_errors.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.projector.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.xdcr_trace.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.debug.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.babysitter.log 2015-06-21 23:12:58 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpp6a5bT/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.http_access.log uname (uname -a) - OK time and TZ (date; date -u) - OK raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK raw /etc/timezone (cat /etc/timezone) - Exit code 1 System Hardware (lshw -json || lshw) - Exit code 127 Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK Raw /proc/vmstat (cat /proc/vmstat) - OK Raw /proc/mounts (cat /proc/mounts) - OK Raw /proc/partitions (cat /proc/partitions) - OK Raw /proc/diskstats (cat /proc/diskstats) - OK Raw /proc/interrupts (cat /proc/interrupts) - OK Swap configuration (free -t) - OK Swap configuration (swapon -s) - OK Kernel modules (lsmod) - OK Distro version (cat /etc/redhat-release) - OK Distro version (lsb_release -a) - Exit code 127 Distro version (cat /etc/SuSE-release) - Exit code 1 Distro version (cat /etc/issue) - OK Installed software (rpm -qa) - OK Installed software (COLUMNS=300 dpkg -l) - Exit code 127 Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK sysctl settings (sysctl -a) - OK relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 LVM info (lvdisplay) - OK LVM info (vgdisplay) - OK LVM info (pvdisplay) - OK Network configuration (ifconfig -a) - OK Taking sample 2 after 10.000000 seconds - OK Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 Raw /proc/net/dev (cat /proc/net/dev) - OK Network link statistics (ip -s link) - OK Network status (netstat -anp || netstat -an) - OK Network routing table (netstat -rn) - OK Network socket statistics (ss -an) - OK Arp cache (arp -na) - OK Iptables dump (iptables-save) - OK Raw /etc/hosts (cat /etc/hosts) - OK Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK Filesystem (df -ha) - OK System activity reporter (sar 1 10) - Exit code 127 System paging activity (vmstat 1 10) - OK System uptime (uptime) - OK couchbase user definition (getent passwd couchbase) - OK couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 Interrupt status (intrstat 1 10) - Exit code 127 Processor status (mpstat 1 10) - Exit code 127 System log (cat /var/adm/messages) - Exit code 1 Raw /proc/uptime (cat /proc/uptime) - OK All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK NUMA data (numactl --hardware) - Exit code 127 NUMA data (numactl --show) - Exit code 127 NUMA data (cat /sys/devices/system/node/node*/numastat) - OK Kernel log buffer (dmesg) - OK Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 Network statistics (netstat -s) - OK Full raw netstat (cat /proc/net/netstat) - OK CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... product diag header () - OK Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Version file (cat '/opt/couchbase/VERSION.txt') - OK Manifest file (cat '/opt/couchbase/manifest.txt') - OK Manifest file (cat '/opt/couchbase/manifest.xml') - OK Couchbase config () - OK Couchbase static config () - OK Raw ns_log () - OK Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK couchbase logs (info.log) (cbbrowse_logs info.log) - OK couchbase logs (error.log) (cbbrowse_logs error.log) - OK couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK couchbase logs (views.log) (cbbrowse_logs views.log) - OK couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK couchbase logs (query.log) (cbbrowse_logs query.log) - OK couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '89c1aede6ae100e6f7fc97084970baad']) - OK memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '89c1aede6ae100e6f7fc97084970baad', 'connections']) - OK replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 mctimings (['mctimings', '-u', '_admin', '-P', '89c1aede6ae100e6f7fc97084970baad', '-h', '127.0.0.1:11210', '-v']) - OK adding: /tmp/tmpp6a5bT/couchbase.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/couchbase.log adding: /tmp/tmpp6a5bT/ddocs.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ddocs.log adding: /tmp/tmpp6a5bT/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.error.log adding: /tmp/tmpp6a5bT/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.ns_couchdb.log adding: /tmp/tmpp6a5bT/stats.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/stats.log adding: /tmp/tmpp6a5bT/memcached.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/memcached.log adding: /tmp/tmpp6a5bT/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.ssl_proxy.log adding: /tmp/tmpp6a5bT/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.xdcr_errors.log adding: /tmp/tmpp6a5bT/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.http_access_internal.log adding: /tmp/tmpp6a5bT/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/syslog.tar.gz adding: /tmp/tmpp6a5bT/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.reports.log adding: /tmp/tmpp6a5bT/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.query.log adding: /tmp/tmpp6a5bT/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.couchdb.log adding: /tmp/tmpp6a5bT/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.goxdcr.log adding: /tmp/tmpp6a5bT/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.indexer.log adding: /tmp/tmpp6a5bT/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.info.log adding: /tmp/tmpp6a5bT/diag.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/diag.log adding: /tmp/tmpp6a5bT/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.stats.log adding: /tmp/tmpp6a5bT/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.xdcr.log adding: /tmp/tmpp6a5bT/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.views.log adding: /tmp/tmpp6a5bT/ini.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ini.log adding: /tmp/tmpp6a5bT/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.mapreduce_errors.log adding: /tmp/tmpp6a5bT/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.projector.log adding: /tmp/tmpp6a5bT/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.xdcr_trace.log adding: /tmp/tmpp6a5bT/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.debug.log adding: /tmp/tmpp6a5bT/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.babysitter.log adding: /tmp/tmpp6a5bT/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.107_20150622-060534/ns_server.http_access.log Downloading zipped logs from 172.23.106.107 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: rm -f /root/172.23.106.107-20150621-2310-diag.zip 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.106.108 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.108 with username : root password : couchbase ssh_key: 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.108 Collecting logs from 172.23.106.108 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: cat /proc/cpuinfo 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: df -Th 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: cat /proc/meminfo 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: hostname 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: hostname -d 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:12:59 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: /opt/couchbase/bin/cbcollect_info 172.23.106.108-20150621-2312-diag.zip 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] uname (uname -a) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] time and TZ (date; date -u) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/timezone (cat /etc/timezone) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System Hardware (lshw -json || lshw) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/vmstat (cat /proc/vmstat) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/mounts (cat /proc/mounts) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/partitions (cat /proc/partitions) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/diskstats (cat /proc/diskstats) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/interrupts (cat /proc/interrupts) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (free -t) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (swapon -s) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel modules (lsmod) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/redhat-release) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (lsb_release -a) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/SuSE-release) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/issue) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (rpm -qa) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (COLUMNS=300 dpkg -l) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] sysctl settings (sysctl -a) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (lvdisplay) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (vgdisplay) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (pvdisplay) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (ifconfig -a) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Taking sample 2 after 10.000000 seconds - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/net/dev (cat /proc/net/dev) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network link statistics (ip -s link) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network status (netstat -anp || netstat -an) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network routing table (netstat -rn) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network socket statistics (ss -an) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Arp cache (arp -na) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Iptables dump (iptables-save) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/hosts (cat /etc/hosts) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Filesystem (df -ha) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System activity reporter (sar 1 10) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System paging activity (vmstat 1 10) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System uptime (uptime) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user definition (getent passwd couchbase) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Interrupt status (intrstat 1 10) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processor status (mpstat 1 10) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System log (cat /var/adm/messages) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/uptime (cat /proc/uptime) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --hardware) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --show) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (cat /sys/devices/system/node/node*/numastat) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel log buffer (dmesg) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network statistics (netstat -s) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Full raw netstat (cat /proc/net/netstat) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] product diag header () - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Version file (cat '/opt/couchbase/VERSION.txt') - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.txt') - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.xml') - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase config () - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase static config () - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw ns_log () - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (info.log) (cbbrowse_logs info.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (error.log) (cbbrowse_logs error.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (views.log) (cbbrowse_logs views.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', 'a85d42199112ce7517f61dde3f800939', 'connections']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] mctimings (['mctimings', '-u', '_admin', '-P', 'a85d42199112ce7517f61dde3f800939', '-h', '127.0.0.1:11210', '-v']) - OK 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/couchbase.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/couchbase.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ddocs.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ddocs.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.error.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.ns_couchdb.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/stats.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/stats.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/memcached.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/memcached.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.ssl_proxy.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.xdcr_errors.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.http_access_internal.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/syslog.tar.gz 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.reports.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.query.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.couchdb.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.goxdcr.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.indexer.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.info.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/diag.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/diag.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.stats.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.xdcr.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.views.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ini.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ini.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.mapreduce_errors.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.projector.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.xdcr_trace.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.debug.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.babysitter.log 2015-06-21 23:14:07 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpq_m2aA/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.http_access.log uname (uname -a) - OK time and TZ (date; date -u) - OK raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK raw /etc/timezone (cat /etc/timezone) - Exit code 1 System Hardware (lshw -json || lshw) - Exit code 127 Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK Raw /proc/vmstat (cat /proc/vmstat) - OK Raw /proc/mounts (cat /proc/mounts) - OK Raw /proc/partitions (cat /proc/partitions) - OK Raw /proc/diskstats (cat /proc/diskstats) - OK Raw /proc/interrupts (cat /proc/interrupts) - OK Swap configuration (free -t) - OK Swap configuration (swapon -s) - OK Kernel modules (lsmod) - OK Distro version (cat /etc/redhat-release) - OK Distro version (lsb_release -a) - Exit code 127 Distro version (cat /etc/SuSE-release) - Exit code 1 Distro version (cat /etc/issue) - OK Installed software (rpm -qa) - OK Installed software (COLUMNS=300 dpkg -l) - Exit code 127 Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK sysctl settings (sysctl -a) - OK relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 LVM info (lvdisplay) - OK LVM info (vgdisplay) - OK LVM info (pvdisplay) - OK Network configuration (ifconfig -a) - OK Taking sample 2 after 10.000000 seconds - OK Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 Raw /proc/net/dev (cat /proc/net/dev) - OK Network link statistics (ip -s link) - OK Network status (netstat -anp || netstat -an) - OK Network routing table (netstat -rn) - OK Network socket statistics (ss -an) - OK Arp cache (arp -na) - OK Iptables dump (iptables-save) - OK Raw /etc/hosts (cat /etc/hosts) - OK Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK Filesystem (df -ha) - OK System activity reporter (sar 1 10) - Exit code 127 System paging activity (vmstat 1 10) - OK System uptime (uptime) - OK couchbase user definition (getent passwd couchbase) - OK couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 Interrupt status (intrstat 1 10) - Exit code 127 Processor status (mpstat 1 10) - Exit code 127 System log (cat /var/adm/messages) - Exit code 1 Raw /proc/uptime (cat /proc/uptime) - OK All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK NUMA data (numactl --hardware) - Exit code 127 NUMA data (numactl --show) - Exit code 127 NUMA data (cat /sys/devices/system/node/node*/numastat) - OK Kernel log buffer (dmesg) - OK Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 Network statistics (netstat -s) - OK Full raw netstat (cat /proc/net/netstat) - OK CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... product diag header () - OK Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Version file (cat '/opt/couchbase/VERSION.txt') - OK Manifest file (cat '/opt/couchbase/manifest.txt') - OK Manifest file (cat '/opt/couchbase/manifest.xml') - OK Couchbase config () - OK Couchbase static config () - OK Raw ns_log () - OK Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK couchbase logs (info.log) (cbbrowse_logs info.log) - OK couchbase logs (error.log) (cbbrowse_logs error.log) - OK couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK couchbase logs (views.log) (cbbrowse_logs views.log) - OK couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', 'a85d42199112ce7517f61dde3f800939']) - OK memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', 'a85d42199112ce7517f61dde3f800939', 'connections']) - OK replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 mctimings (['mctimings', '-u', '_admin', '-P', 'a85d42199112ce7517f61dde3f800939', '-h', '127.0.0.1:11210', '-v']) - OK adding: /tmp/tmpq_m2aA/couchbase.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/couchbase.log adding: /tmp/tmpq_m2aA/ddocs.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ddocs.log adding: /tmp/tmpq_m2aA/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.error.log adding: /tmp/tmpq_m2aA/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.ns_couchdb.log adding: /tmp/tmpq_m2aA/stats.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/stats.log adding: /tmp/tmpq_m2aA/memcached.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/memcached.log adding: /tmp/tmpq_m2aA/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.ssl_proxy.log adding: /tmp/tmpq_m2aA/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.xdcr_errors.log adding: /tmp/tmpq_m2aA/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.http_access_internal.log adding: /tmp/tmpq_m2aA/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/syslog.tar.gz adding: /tmp/tmpq_m2aA/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.reports.log adding: /tmp/tmpq_m2aA/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.query.log adding: /tmp/tmpq_m2aA/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.couchdb.log adding: /tmp/tmpq_m2aA/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.goxdcr.log adding: /tmp/tmpq_m2aA/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.indexer.log adding: /tmp/tmpq_m2aA/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.info.log adding: /tmp/tmpq_m2aA/diag.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/diag.log adding: /tmp/tmpq_m2aA/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.stats.log adding: /tmp/tmpq_m2aA/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.xdcr.log adding: /tmp/tmpq_m2aA/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.views.log adding: /tmp/tmpq_m2aA/ini.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ini.log adding: /tmp/tmpq_m2aA/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.mapreduce_errors.log adding: /tmp/tmpq_m2aA/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.projector.log adding: /tmp/tmpq_m2aA/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.xdcr_trace.log adding: /tmp/tmpq_m2aA/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.debug.log adding: /tmp/tmpq_m2aA/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.babysitter.log adding: /tmp/tmpq_m2aA/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.108_20150622-060907/ns_server.http_access.log Downloading zipped logs from 172.23.106.108 2015-06-21 23:14:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: rm -f /root/172.23.106.108-20150621-2312-diag.zip 2015-06-21 23:14:08 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.106.110 2015-06-21 23:14:08 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.110 with username : root password : couchbase ssh_key: 2015-06-21 23:14:08 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.110 Collecting logs from 172.23.106.110 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: cat /proc/cpuinfo 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: df -Th 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: cat /proc/meminfo 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: hostname 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: hostname -d 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:14:09 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: /opt/couchbase/bin/cbcollect_info 172.23.106.110-20150621-2314-diag.zip 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] uname (uname -a) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] time and TZ (date; date -u) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/timezone (cat /etc/timezone) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System Hardware (lshw -json || lshw) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/vmstat (cat /proc/vmstat) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/mounts (cat /proc/mounts) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/partitions (cat /proc/partitions) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/diskstats (cat /proc/diskstats) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/interrupts (cat /proc/interrupts) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (free -t) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (swapon -s) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel modules (lsmod) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/redhat-release) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (lsb_release -a) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/SuSE-release) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/issue) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (rpm -qa) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (COLUMNS=300 dpkg -l) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] sysctl settings (sysctl -a) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (lvdisplay) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (vgdisplay) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (pvdisplay) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (ifconfig -a) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Taking sample 2 after 10.000000 seconds - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/net/dev (cat /proc/net/dev) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network link statistics (ip -s link) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network status (netstat -anp || netstat -an) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network routing table (netstat -rn) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network socket statistics (ss -an) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Arp cache (arp -na) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Iptables dump (iptables-save) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/hosts (cat /etc/hosts) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Filesystem (df -ha) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System activity reporter (sar 1 10) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System paging activity (vmstat 1 10) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System uptime (uptime) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user definition (getent passwd couchbase) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Interrupt status (intrstat 1 10) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processor status (mpstat 1 10) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System log (cat /var/adm/messages) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/uptime (cat /proc/uptime) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --hardware) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --show) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (cat /sys/devices/system/node/node*/numastat) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel log buffer (dmesg) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network statistics (netstat -s) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Full raw netstat (cat /proc/net/netstat) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] product diag header () - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Version file (cat '/opt/couchbase/VERSION.txt') - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.txt') - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.xml') - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase config () - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase static config () - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw ns_log () - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (info.log) (cbbrowse_logs info.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (error.log) (cbbrowse_logs error.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (views.log) (cbbrowse_logs views.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '3db5104eaac23776a2e6e721b853200a', 'connections']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] mctimings (['mctimings', '-u', '_admin', '-P', '3db5104eaac23776a2e6e721b853200a', '-h', '127.0.0.1:11210', '-v']) - OK 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/couchbase.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/couchbase.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ddocs.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ddocs.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.error.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.ns_couchdb.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/stats.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/stats.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/memcached.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/memcached.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.ssl_proxy.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.xdcr_errors.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.http_access_internal.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/syslog.tar.gz 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.reports.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.query.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.couchdb.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.goxdcr.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.indexer.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.info.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/diag.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/diag.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.stats.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.xdcr.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.views.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ini.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ini.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.mapreduce_errors.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.projector.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.xdcr_trace.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.debug.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.babysitter.log 2015-06-21 23:15:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp551BYw/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.http_access.log uname (uname -a) - OK time and TZ (date; date -u) - OK raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK raw /etc/timezone (cat /etc/timezone) - Exit code 1 System Hardware (lshw -json || lshw) - Exit code 127 Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK Raw /proc/vmstat (cat /proc/vmstat) - OK Raw /proc/mounts (cat /proc/mounts) - OK Raw /proc/partitions (cat /proc/partitions) - OK Raw /proc/diskstats (cat /proc/diskstats) - OK Raw /proc/interrupts (cat /proc/interrupts) - OK Swap configuration (free -t) - OK Swap configuration (swapon -s) - OK Kernel modules (lsmod) - OK Distro version (cat /etc/redhat-release) - OK Distro version (lsb_release -a) - Exit code 127 Distro version (cat /etc/SuSE-release) - Exit code 1 Distro version (cat /etc/issue) - OK Installed software (rpm -qa) - OK Installed software (COLUMNS=300 dpkg -l) - Exit code 127 Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK sysctl settings (sysctl -a) - OK relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 LVM info (lvdisplay) - OK LVM info (vgdisplay) - OK LVM info (pvdisplay) - OK Network configuration (ifconfig -a) - OK Taking sample 2 after 10.000000 seconds - OK Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 Raw /proc/net/dev (cat /proc/net/dev) - OK Network link statistics (ip -s link) - OK Network status (netstat -anp || netstat -an) - OK Network routing table (netstat -rn) - OK Network socket statistics (ss -an) - OK Arp cache (arp -na) - OK Iptables dump (iptables-save) - OK Raw /etc/hosts (cat /etc/hosts) - OK Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK Filesystem (df -ha) - OK System activity reporter (sar 1 10) - Exit code 127 System paging activity (vmstat 1 10) - OK System uptime (uptime) - OK couchbase user definition (getent passwd couchbase) - OK couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 Interrupt status (intrstat 1 10) - Exit code 127 Processor status (mpstat 1 10) - Exit code 127 System log (cat /var/adm/messages) - Exit code 1 Raw /proc/uptime (cat /proc/uptime) - OK All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK NUMA data (numactl --hardware) - Exit code 127 NUMA data (numactl --show) - Exit code 127 NUMA data (cat /sys/devices/system/node/node*/numastat) - OK Kernel log buffer (dmesg) - OK Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 Network statistics (netstat -s) - OK Full raw netstat (cat /proc/net/netstat) - OK CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... product diag header () - OK Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Version file (cat '/opt/couchbase/VERSION.txt') - OK Manifest file (cat '/opt/couchbase/manifest.txt') - OK Manifest file (cat '/opt/couchbase/manifest.xml') - OK Couchbase config () - OK Couchbase static config () - OK Raw ns_log () - OK Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK couchbase logs (info.log) (cbbrowse_logs info.log) - OK couchbase logs (error.log) (cbbrowse_logs error.log) - OK couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK couchbase logs (views.log) (cbbrowse_logs views.log) - OK couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '3db5104eaac23776a2e6e721b853200a']) - OK memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '3db5104eaac23776a2e6e721b853200a', 'connections']) - OK replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 mctimings (['mctimings', '-u', '_admin', '-P', '3db5104eaac23776a2e6e721b853200a', '-h', '127.0.0.1:11210', '-v']) - OK adding: /tmp/tmp551BYw/couchbase.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/couchbase.log adding: /tmp/tmp551BYw/ddocs.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ddocs.log adding: /tmp/tmp551BYw/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.error.log adding: /tmp/tmp551BYw/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.ns_couchdb.log adding: /tmp/tmp551BYw/stats.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/stats.log adding: /tmp/tmp551BYw/memcached.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/memcached.log adding: /tmp/tmp551BYw/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.ssl_proxy.log adding: /tmp/tmp551BYw/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.xdcr_errors.log adding: /tmp/tmp551BYw/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.http_access_internal.log adding: /tmp/tmp551BYw/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/syslog.tar.gz adding: /tmp/tmp551BYw/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.reports.log adding: /tmp/tmp551BYw/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.query.log adding: /tmp/tmp551BYw/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.couchdb.log adding: /tmp/tmp551BYw/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.goxdcr.log adding: /tmp/tmp551BYw/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.indexer.log adding: /tmp/tmp551BYw/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.info.log adding: /tmp/tmp551BYw/diag.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/diag.log adding: /tmp/tmp551BYw/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.stats.log adding: /tmp/tmp551BYw/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.xdcr.log adding: /tmp/tmp551BYw/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.views.log adding: /tmp/tmp551BYw/ini.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ini.log adding: /tmp/tmp551BYw/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.mapreduce_errors.log adding: /tmp/tmp551BYw/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.projector.log adding: /tmp/tmp551BYw/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.xdcr_trace.log adding: /tmp/tmp551BYw/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.debug.log adding: /tmp/tmp551BYw/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.babysitter.log adding: /tmp/tmp551BYw/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.110_20150622-060934/ns_server.http_access.log Downloading zipped logs from 172.23.106.110 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: rm -f /root/172.23.106.110-20150621-2314-diag.zip 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.106.111 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.111 with username : root password : couchbase ssh_key: 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.111 Collecting logs from 172.23.106.111 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: cat /proc/cpuinfo 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: df -Th 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: cat /proc/meminfo 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: hostname 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: hostname -d 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:15:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: /opt/couchbase/bin/cbcollect_info 172.23.106.111-20150621-2315-diag.zip 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] uname (uname -a) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] time and TZ (date; date -u) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/timezone (cat /etc/timezone) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System Hardware (lshw -json || lshw) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/vmstat (cat /proc/vmstat) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/mounts (cat /proc/mounts) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/partitions (cat /proc/partitions) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/diskstats (cat /proc/diskstats) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/interrupts (cat /proc/interrupts) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (free -t) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (swapon -s) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel modules (lsmod) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/redhat-release) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (lsb_release -a) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/SuSE-release) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/issue) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (rpm -qa) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (COLUMNS=300 dpkg -l) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] sysctl settings (sysctl -a) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (lvdisplay) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (vgdisplay) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (pvdisplay) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (ifconfig -a) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Taking sample 2 after 10.000000 seconds - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/net/dev (cat /proc/net/dev) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network link statistics (ip -s link) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network status (netstat -anp || netstat -an) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network routing table (netstat -rn) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network socket statistics (ss -an) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Arp cache (arp -na) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Iptables dump (iptables-save) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/hosts (cat /etc/hosts) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Filesystem (df -ha) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System activity reporter (sar 1 10) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System paging activity (vmstat 1 10) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System uptime (uptime) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user definition (getent passwd couchbase) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Interrupt status (intrstat 1 10) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processor status (mpstat 1 10) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System log (cat /var/adm/messages) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/uptime (cat /proc/uptime) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --hardware) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --show) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (cat /sys/devices/system/node/node*/numastat) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel log buffer (dmesg) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network statistics (netstat -s) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Full raw netstat (cat /proc/net/netstat) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] product diag header () - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Version file (cat '/opt/couchbase/VERSION.txt') - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.txt') - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.xml') - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase config () - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase static config () - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw ns_log () - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (info.log) (cbbrowse_logs info.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (error.log) (cbbrowse_logs error.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (views.log) (cbbrowse_logs views.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', 'db79e624ead06317c93f04dfc14ffb30', 'connections']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] mctimings (['mctimings', '-u', '_admin', '-P', 'db79e624ead06317c93f04dfc14ffb30', '-h', '127.0.0.1:11210', '-v']) - OK 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/couchbase.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/couchbase.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ddocs.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ddocs.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.error.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.ns_couchdb.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/stats.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/stats.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/memcached.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/memcached.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.ssl_proxy.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.xdcr_errors.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.http_access_internal.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/syslog.tar.gz 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.reports.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.query.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.couchdb.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.goxdcr.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.indexer.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.info.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/diag.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/diag.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.stats.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.xdcr.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.views.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ini.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ini.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.mapreduce_errors.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.projector.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.xdcr_trace.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.debug.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.babysitter.log 2015-06-21 23:16:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmpXhTTFq/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.http_access.log uname (uname -a) - OK time and TZ (date; date -u) - OK raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK raw /etc/timezone (cat /etc/timezone) - Exit code 1 System Hardware (lshw -json || lshw) - Exit code 127 Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK Raw /proc/vmstat (cat /proc/vmstat) - OK Raw /proc/mounts (cat /proc/mounts) - OK Raw /proc/partitions (cat /proc/partitions) - OK Raw /proc/diskstats (cat /proc/diskstats) - OK Raw /proc/interrupts (cat /proc/interrupts) - OK Swap configuration (free -t) - OK Swap configuration (swapon -s) - OK Kernel modules (lsmod) - OK Distro version (cat /etc/redhat-release) - OK Distro version (lsb_release -a) - Exit code 127 Distro version (cat /etc/SuSE-release) - Exit code 1 Distro version (cat /etc/issue) - OK Installed software (rpm -qa) - OK Installed software (COLUMNS=300 dpkg -l) - Exit code 127 Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK sysctl settings (sysctl -a) - OK relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 LVM info (lvdisplay) - OK LVM info (vgdisplay) - OK LVM info (pvdisplay) - OK Network configuration (ifconfig -a) - OK Taking sample 2 after 10.000000 seconds - OK Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 Raw /proc/net/dev (cat /proc/net/dev) - OK Network link statistics (ip -s link) - OK Network status (netstat -anp || netstat -an) - OK Network routing table (netstat -rn) - OK Network socket statistics (ss -an) - OK Arp cache (arp -na) - OK Iptables dump (iptables-save) - OK Raw /etc/hosts (cat /etc/hosts) - OK Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK Filesystem (df -ha) - OK System activity reporter (sar 1 10) - Exit code 127 System paging activity (vmstat 1 10) - OK System uptime (uptime) - OK couchbase user definition (getent passwd couchbase) - OK couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 Interrupt status (intrstat 1 10) - Exit code 127 Processor status (mpstat 1 10) - Exit code 127 System log (cat /var/adm/messages) - Exit code 1 Raw /proc/uptime (cat /proc/uptime) - OK All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK NUMA data (numactl --hardware) - Exit code 127 NUMA data (numactl --show) - Exit code 127 NUMA data (cat /sys/devices/system/node/node*/numastat) - OK Kernel log buffer (dmesg) - OK Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 Network statistics (netstat -s) - OK Full raw netstat (cat /proc/net/netstat) - OK CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... product diag header () - OK Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Version file (cat '/opt/couchbase/VERSION.txt') - OK Manifest file (cat '/opt/couchbase/manifest.txt') - OK Manifest file (cat '/opt/couchbase/manifest.xml') - OK Couchbase config () - OK Couchbase static config () - OK Raw ns_log () - OK Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK couchbase logs (info.log) (cbbrowse_logs info.log) - OK couchbase logs (error.log) (cbbrowse_logs error.log) - OK couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK couchbase logs (views.log) (cbbrowse_logs views.log) - OK couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', 'db79e624ead06317c93f04dfc14ffb30']) - OK memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', 'db79e624ead06317c93f04dfc14ffb30', 'connections']) - OK replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 mctimings (['mctimings', '-u', '_admin', '-P', 'db79e624ead06317c93f04dfc14ffb30', '-h', '127.0.0.1:11210', '-v']) - OK adding: /tmp/tmpXhTTFq/couchbase.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/couchbase.log adding: /tmp/tmpXhTTFq/ddocs.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ddocs.log adding: /tmp/tmpXhTTFq/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.error.log adding: /tmp/tmpXhTTFq/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.ns_couchdb.log adding: /tmp/tmpXhTTFq/stats.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/stats.log adding: /tmp/tmpXhTTFq/memcached.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/memcached.log adding: /tmp/tmpXhTTFq/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.ssl_proxy.log adding: /tmp/tmpXhTTFq/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.xdcr_errors.log adding: /tmp/tmpXhTTFq/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.http_access_internal.log adding: /tmp/tmpXhTTFq/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/syslog.tar.gz adding: /tmp/tmpXhTTFq/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.reports.log adding: /tmp/tmpXhTTFq/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.query.log adding: /tmp/tmpXhTTFq/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.couchdb.log adding: /tmp/tmpXhTTFq/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.goxdcr.log adding: /tmp/tmpXhTTFq/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.indexer.log adding: /tmp/tmpXhTTFq/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.info.log adding: /tmp/tmpXhTTFq/diag.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/diag.log adding: /tmp/tmpXhTTFq/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.stats.log adding: /tmp/tmpXhTTFq/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.xdcr.log adding: /tmp/tmpXhTTFq/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.views.log adding: /tmp/tmpXhTTFq/ini.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ini.log adding: /tmp/tmpXhTTFq/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.mapreduce_errors.log adding: /tmp/tmpXhTTFq/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.projector.log adding: /tmp/tmpXhTTFq/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.xdcr_trace.log adding: /tmp/tmpXhTTFq/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.debug.log adding: /tmp/tmpXhTTFq/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.babysitter.log adding: /tmp/tmpXhTTFq/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.111_20150622-062047/ns_server.http_access.log Downloading zipped logs from 172.23.106.111 2015-06-21 23:16:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: rm -f /root/172.23.106.111-20150621-2315-diag.zip 2015-06-21 23:16:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.106.112 2015-06-21 23:16:16 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.112 with username : root password : couchbase ssh_key: 2015-06-21 23:16:16 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.112 Collecting logs from 172.23.106.112 2015-06-21 23:16:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: cat /proc/cpuinfo 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: df -Th 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: cat /proc/meminfo 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: hostname 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: hostname -d 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:16:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: /opt/couchbase/bin/cbcollect_info 172.23.106.112-20150621-2316-diag.zip 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] uname (uname -a) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] time and TZ (date; date -u) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/timezone (cat /etc/timezone) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System Hardware (lshw -json || lshw) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/vmstat (cat /proc/vmstat) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/mounts (cat /proc/mounts) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/partitions (cat /proc/partitions) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/diskstats (cat /proc/diskstats) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/interrupts (cat /proc/interrupts) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (free -t) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (swapon -s) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel modules (lsmod) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/redhat-release) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (lsb_release -a) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/SuSE-release) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/issue) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (rpm -qa) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (COLUMNS=300 dpkg -l) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] sysctl settings (sysctl -a) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (lvdisplay) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (vgdisplay) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (pvdisplay) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (ifconfig -a) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Taking sample 2 after 10.000000 seconds - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/net/dev (cat /proc/net/dev) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network link statistics (ip -s link) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network status (netstat -anp || netstat -an) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network routing table (netstat -rn) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network socket statistics (ss -an) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Arp cache (arp -na) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Iptables dump (iptables-save) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/hosts (cat /etc/hosts) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Filesystem (df -ha) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System activity reporter (sar 1 10) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System paging activity (vmstat 1 10) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System uptime (uptime) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user definition (getent passwd couchbase) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Interrupt status (intrstat 1 10) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processor status (mpstat 1 10) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System log (cat /var/adm/messages) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/uptime (cat /proc/uptime) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --hardware) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --show) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (cat /sys/devices/system/node/node*/numastat) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel log buffer (dmesg) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network statistics (netstat -s) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Full raw netstat (cat /proc/net/netstat) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] product diag header () - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Version file (cat '/opt/couchbase/VERSION.txt') - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.txt') - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.xml') - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase config () - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase static config () - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw ns_log () - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (info.log) (cbbrowse_logs info.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (error.log) (cbbrowse_logs error.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (views.log) (cbbrowse_logs views.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '3f883233e2bc1f82cb64d0dd89d4d286', 'connections']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] mctimings (['mctimings', '-u', '_admin', '-P', '3f883233e2bc1f82cb64d0dd89d4d286', '-h', '127.0.0.1:11210', '-v']) - OK 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/couchbase.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/couchbase.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ddocs.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ddocs.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.error.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.ns_couchdb.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/stats.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/stats.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/memcached.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/memcached.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.ssl_proxy.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.xdcr_errors.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.http_access_internal.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/syslog.tar.gz 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.reports.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.query.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.couchdb.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.goxdcr.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.indexer.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.info.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/diag.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/diag.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.stats.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.xdcr.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.views.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ini.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ini.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.mapreduce_errors.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.projector.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.xdcr_trace.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.debug.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.babysitter.log 2015-06-21 23:17:15 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp5YQRTJ/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.http_access.log uname (uname -a) - OK time and TZ (date; date -u) - OK raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK raw /etc/timezone (cat /etc/timezone) - Exit code 1 System Hardware (lshw -json || lshw) - Exit code 127 Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK Raw /proc/vmstat (cat /proc/vmstat) - OK Raw /proc/mounts (cat /proc/mounts) - OK Raw /proc/partitions (cat /proc/partitions) - OK Raw /proc/diskstats (cat /proc/diskstats) - OK Raw /proc/interrupts (cat /proc/interrupts) - OK Swap configuration (free -t) - OK Swap configuration (swapon -s) - OK Kernel modules (lsmod) - OK Distro version (cat /etc/redhat-release) - OK Distro version (lsb_release -a) - Exit code 127 Distro version (cat /etc/SuSE-release) - Exit code 1 Distro version (cat /etc/issue) - OK Installed software (rpm -qa) - OK Installed software (COLUMNS=300 dpkg -l) - Exit code 127 Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK sysctl settings (sysctl -a) - OK relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 LVM info (lvdisplay) - OK LVM info (vgdisplay) - OK LVM info (pvdisplay) - OK Network configuration (ifconfig -a) - OK Taking sample 2 after 10.000000 seconds - OK Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 Raw /proc/net/dev (cat /proc/net/dev) - OK Network link statistics (ip -s link) - OK Network status (netstat -anp || netstat -an) - OK Network routing table (netstat -rn) - OK Network socket statistics (ss -an) - OK Arp cache (arp -na) - OK Iptables dump (iptables-save) - OK Raw /etc/hosts (cat /etc/hosts) - OK Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK Filesystem (df -ha) - OK System activity reporter (sar 1 10) - Exit code 127 System paging activity (vmstat 1 10) - OK System uptime (uptime) - OK couchbase user definition (getent passwd couchbase) - OK couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 Interrupt status (intrstat 1 10) - Exit code 127 Processor status (mpstat 1 10) - Exit code 127 System log (cat /var/adm/messages) - Exit code 1 Raw /proc/uptime (cat /proc/uptime) - OK All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK NUMA data (numactl --hardware) - Exit code 127 NUMA data (numactl --show) - Exit code 127 NUMA data (cat /sys/devices/system/node/node*/numastat) - OK Kernel log buffer (dmesg) - OK Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 Network statistics (netstat -s) - OK Full raw netstat (cat /proc/net/netstat) - OK CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... product diag header () - OK Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Version file (cat '/opt/couchbase/VERSION.txt') - OK Manifest file (cat '/opt/couchbase/manifest.txt') - OK Manifest file (cat '/opt/couchbase/manifest.xml') - OK Couchbase config () - OK Couchbase static config () - OK Raw ns_log () - OK Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK couchbase logs (info.log) (cbbrowse_logs info.log) - OK couchbase logs (error.log) (cbbrowse_logs error.log) - OK couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK couchbase logs (views.log) (cbbrowse_logs views.log) - OK couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - OK memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '3f883233e2bc1f82cb64d0dd89d4d286']) - OK memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '3f883233e2bc1f82cb64d0dd89d4d286', 'connections']) - OK replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 mctimings (['mctimings', '-u', '_admin', '-P', '3f883233e2bc1f82cb64d0dd89d4d286', '-h', '127.0.0.1:11210', '-v']) - OK adding: /tmp/tmp5YQRTJ/couchbase.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/couchbase.log adding: /tmp/tmp5YQRTJ/ddocs.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ddocs.log adding: /tmp/tmp5YQRTJ/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.error.log adding: /tmp/tmp5YQRTJ/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.ns_couchdb.log adding: /tmp/tmp5YQRTJ/stats.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/stats.log adding: /tmp/tmp5YQRTJ/memcached.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/memcached.log adding: /tmp/tmp5YQRTJ/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.ssl_proxy.log adding: /tmp/tmp5YQRTJ/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.xdcr_errors.log adding: /tmp/tmp5YQRTJ/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.http_access_internal.log adding: /tmp/tmp5YQRTJ/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/syslog.tar.gz adding: /tmp/tmp5YQRTJ/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.reports.log adding: /tmp/tmp5YQRTJ/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.query.log adding: /tmp/tmp5YQRTJ/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.couchdb.log adding: /tmp/tmp5YQRTJ/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.goxdcr.log adding: /tmp/tmp5YQRTJ/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.indexer.log adding: /tmp/tmp5YQRTJ/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.info.log adding: /tmp/tmp5YQRTJ/diag.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/diag.log adding: /tmp/tmp5YQRTJ/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.stats.log adding: /tmp/tmp5YQRTJ/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.xdcr.log adding: /tmp/tmp5YQRTJ/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.views.log adding: /tmp/tmp5YQRTJ/ini.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ini.log adding: /tmp/tmp5YQRTJ/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.mapreduce_errors.log adding: /tmp/tmp5YQRTJ/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.projector.log adding: /tmp/tmp5YQRTJ/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.xdcr_trace.log adding: /tmp/tmp5YQRTJ/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.debug.log adding: /tmp/tmp5YQRTJ/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.babysitter.log adding: /tmp/tmp5YQRTJ/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.112_20150622-062148/ns_server.http_access.log Downloading zipped logs from 172.23.106.112 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: rm -f /root/172.23.106.112-20150621-2316-diag.zip 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.106.109 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.109 with username : root password : couchbase ssh_key: 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.109 Collecting logs from 172.23.106.109 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: cat /proc/cpuinfo 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: df -Th 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: cat /proc/meminfo 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: hostname 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: hostname -d 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:17:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: /opt/couchbase/bin/cbcollect_info 172.23.106.109-20150621-2317-diag.zip 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] uname (uname -a) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] time and TZ (date; date -u) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] raw /etc/timezone (cat /etc/timezone) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System Hardware (lshw -json || lshw) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/vmstat (cat /proc/vmstat) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/mounts (cat /proc/mounts) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/partitions (cat /proc/partitions) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/diskstats (cat /proc/diskstats) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/interrupts (cat /proc/interrupts) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (free -t) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Swap configuration (swapon -s) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel modules (lsmod) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/redhat-release) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (lsb_release -a) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/SuSE-release) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Distro version (cat /etc/issue) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (rpm -qa) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Installed software (COLUMNS=300 dpkg -l) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] sysctl settings (sysctl -a) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (lvdisplay) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (vgdisplay) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] LVM info (pvdisplay) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (ifconfig -a) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Taking sample 2 after 10.000000 seconds - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/net/dev (cat /proc/net/dev) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network link statistics (ip -s link) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network status (netstat -anp || netstat -an) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network routing table (netstat -rn) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network socket statistics (ss -an) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Arp cache (arp -na) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Iptables dump (iptables-save) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/hosts (cat /etc/hosts) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Filesystem (df -ha) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System activity reporter (sar 1 10) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System paging activity (vmstat 1 10) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System uptime (uptime) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user definition (getent passwd couchbase) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Interrupt status (intrstat 1 10) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processor status (mpstat 1 10) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] System log (cat /var/adm/messages) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw /proc/uptime (cat /proc/uptime) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --hardware) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (numactl --show) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] NUMA data (cat /sys/devices/system/node/node*/numastat) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Kernel log buffer (dmesg) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Network statistics (netstat -s) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Full raw netstat (cat /proc/net/netstat) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] product diag header () - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Version file (cat '/opt/couchbase/VERSION.txt') - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.txt') - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Manifest file (cat '/opt/couchbase/manifest.xml') - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase config () - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Couchbase static config () - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Raw ns_log () - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (info.log) (cbbrowse_logs info.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (error.log) (cbbrowse_logs error.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (views.log) (cbbrowse_logs views.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '172af6a308a2aad39bb37a6c442835af', 'connections']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] mctimings (['mctimings', '-u', '_admin', '-P', '172af6a308a2aad39bb37a6c442835af', '-h', '127.0.0.1:11210', '-v']) - OK 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/couchbase.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/couchbase.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ddocs.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ddocs.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.error.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.ns_couchdb.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/stats.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/stats.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/memcached.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/memcached.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.ssl_proxy.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.xdcr_errors.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.http_access_internal.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/syslog.tar.gz 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.reports.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.query.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.couchdb.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.goxdcr.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.indexer.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.info.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/diag.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/diag.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.stats.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.xdcr.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.views.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ini.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ini.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.mapreduce_errors.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.projector.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.xdcr_trace.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.debug.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.babysitter.log 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.log_command_output] adding: /tmp/tmp3xrvHV/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.http_access.log uname (uname -a) - OK time and TZ (date; date -u) - OK raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK raw /etc/timezone (cat /etc/timezone) - Exit code 1 System Hardware (lshw -json || lshw) - Exit code 127 Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK Raw /proc/vmstat (cat /proc/vmstat) - OK Raw /proc/mounts (cat /proc/mounts) - OK Raw /proc/partitions (cat /proc/partitions) - OK Raw /proc/diskstats (cat /proc/diskstats) - OK Raw /proc/interrupts (cat /proc/interrupts) - OK Swap configuration (free -t) - OK Swap configuration (swapon -s) - OK Kernel modules (lsmod) - OK Distro version (cat /etc/redhat-release) - OK Distro version (lsb_release -a) - Exit code 127 Distro version (cat /etc/SuSE-release) - Exit code 1 Distro version (cat /etc/issue) - OK Installed software (rpm -qa) - OK Installed software (COLUMNS=300 dpkg -l) - Exit code 127 Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK sysctl settings (sysctl -a) - OK relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 LVM info (lvdisplay) - OK LVM info (vgdisplay) - OK LVM info (pvdisplay) - OK Network configuration (ifconfig -a) - OK Taking sample 2 after 10.000000 seconds - OK Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 Raw /proc/net/dev (cat /proc/net/dev) - OK Network link statistics (ip -s link) - OK Network status (netstat -anp || netstat -an) - OK Network routing table (netstat -rn) - OK Network socket statistics (ss -an) - OK Arp cache (arp -na) - OK Iptables dump (iptables-save) - OK Raw /etc/hosts (cat /etc/hosts) - OK Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK Filesystem (df -ha) - OK System activity reporter (sar 1 10) - Exit code 127 System paging activity (vmstat 1 10) - OK System uptime (uptime) - OK couchbase user definition (getent passwd couchbase) - OK couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 Interrupt status (intrstat 1 10) - Exit code 127 Processor status (mpstat 1 10) - Exit code 127 System log (cat /var/adm/messages) - Exit code 1 Raw /proc/uptime (cat /proc/uptime) - OK All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK NUMA data (numactl --hardware) - Exit code 127 NUMA data (numactl --show) - Exit code 127 NUMA data (cat /sys/devices/system/node/node*/numastat) - OK Kernel log buffer (dmesg) - OK Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 Network statistics (netstat -s) - OK Full raw netstat (cat /proc/net/netstat) - OK CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... product diag header () - OK Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK Version file (cat '/opt/couchbase/VERSION.txt') - OK Manifest file (cat '/opt/couchbase/manifest.txt') - OK Manifest file (cat '/opt/couchbase/manifest.xml') - OK Couchbase config () - OK Couchbase static config () - OK Raw ns_log () - OK Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK couchbase diags (curl -sS -u :***** http://127.0.0.1:8091/diag?noLogs=1) - OK couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK couchbase logs (info.log) (cbbrowse_logs info.log) - OK couchbase logs (error.log) (cbbrowse_logs error.log) - OK couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK couchbase logs (views.log) (cbbrowse_logs views.log) - OK couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 memcached stats all (['cbstats', '-a', '127.0.0.1:11210', 'all', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11210', 'checkpoint', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats config (['cbstats', '-a', '127.0.0.1:11210', 'config', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11210', 'dispatcher', 'logs', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats workload (['cbstats', '-a', '127.0.0.1:11210', 'workload', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11210', 'runtimes', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11210', 'scheduler', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11210', 'kvstore', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11210', 'kvtimings', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats tap (['cbstats', '-a', '127.0.0.1:11210', 'tap', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11210', 'tapagg', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats dcp (['cbstats', '-a', '127.0.0.1:11210', 'dcp', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11210', 'dcpagg', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats timings (['cbstats', '-a', '127.0.0.1:11210', 'timings', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats ['raw', 'memory'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'memory', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats ['raw', 'allocator'] (['cbstats', '-a', '127.0.0.1:11210', 'raw', 'allocator', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11210', 'prev-vbucket', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11210', 'vbucket', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11210', 'vbucket-details', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached stats warmup (['cbstats', '-a', '127.0.0.1:11210', 'warmup', '-b', '_admin', '-p', '172af6a308a2aad39bb37a6c442835af']) - OK memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11210', '-u', '_admin', '-P', '172af6a308a2aad39bb37a6c442835af', 'connections']) - OK replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 mctimings (['mctimings', '-u', '_admin', '-P', '172af6a308a2aad39bb37a6c442835af', '-h', '127.0.0.1:11210', '-v']) - OK adding: /tmp/tmp3xrvHV/couchbase.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/couchbase.log adding: /tmp/tmp3xrvHV/ddocs.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ddocs.log adding: /tmp/tmp3xrvHV/ns_server.error.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.error.log adding: /tmp/tmp3xrvHV/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.ns_couchdb.log adding: /tmp/tmp3xrvHV/stats.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/stats.log adding: /tmp/tmp3xrvHV/memcached.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/memcached.log adding: /tmp/tmp3xrvHV/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.ssl_proxy.log adding: /tmp/tmp3xrvHV/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.xdcr_errors.log adding: /tmp/tmp3xrvHV/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.http_access_internal.log adding: /tmp/tmp3xrvHV/syslog.tar.gz -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/syslog.tar.gz adding: /tmp/tmp3xrvHV/ns_server.reports.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.reports.log adding: /tmp/tmp3xrvHV/ns_server.query.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.query.log adding: /tmp/tmp3xrvHV/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.couchdb.log adding: /tmp/tmp3xrvHV/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.goxdcr.log adding: /tmp/tmp3xrvHV/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.indexer.log adding: /tmp/tmp3xrvHV/ns_server.info.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.info.log adding: /tmp/tmp3xrvHV/diag.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/diag.log adding: /tmp/tmp3xrvHV/ns_server.stats.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.stats.log adding: /tmp/tmp3xrvHV/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.xdcr.log adding: /tmp/tmp3xrvHV/ns_server.views.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.views.log adding: /tmp/tmp3xrvHV/ini.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ini.log adding: /tmp/tmp3xrvHV/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.mapreduce_errors.log adding: /tmp/tmp3xrvHV/ns_server.projector.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.projector.log adding: /tmp/tmp3xrvHV/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.xdcr_trace.log adding: /tmp/tmp3xrvHV/ns_server.debug.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.debug.log adding: /tmp/tmp3xrvHV/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.babysitter.log adding: /tmp/tmp3xrvHV/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.106.109_20150622-062247/ns_server.http_access.log Downloading zipped logs from 172.23.106.109 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: rm -f /root/172.23.106.109-20150621-2317-diag.zip 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing core dumps files from 172.23.106.107 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.107 with username : root password : couchbase ssh_key: 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.107 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: cat /proc/cpuinfo 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: df -Th 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:14 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: cat /proc/meminfo 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: hostname 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.107: hostname -d 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully looking for crashes on 172.23.106.107 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.107 grabbing core dumps files from 172.23.106.108 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.108 with username : root password : couchbase ssh_key: 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.108 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: cat /proc/cpuinfo 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: df -Th 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:15 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: cat /proc/meminfo 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: hostname 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.108: hostname -d 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully looking for crashes on 172.23.106.108 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.108 grabbing core dumps files from 172.23.106.110 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.110 with username : root password : couchbase ssh_key: 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.110 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: cat /proc/cpuinfo 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: df -Th 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: cat /proc/meminfo 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: hostname 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.110: hostname -d 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully looking for crashes on 172.23.106.110 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.110 grabbing core dumps files from 172.23.106.111 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.111 with username : root password : couchbase ssh_key: 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.111 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: cat /proc/cpuinfo 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: df -Th 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: cat /proc/meminfo 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: hostname 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.111: hostname -d 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully looking for crashes on 172.23.106.111 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.111 grabbing core dumps files from 172.23.106.112 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.112 with username : root password : couchbase ssh_key: 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.112 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: cat /proc/cpuinfo 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: df -Th 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: cat /proc/meminfo 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: hostname 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.112: hostname -d 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully looking for crashes on 172.23.106.112 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.112 grabbing core dumps files from 172.23.106.109 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.106.109 with username : root password : couchbase ssh_key: 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.106.109 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: cat /proc/cpuinfo 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: df -Th 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: cat /proc/meminfo 2015-06-21 23:18:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: hostname 2015-06-21 23:18:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully 2015-06-21 23:18:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.106.109: hostname -d 2015-06-21 23:18:20 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully looking for crashes on 172.23.106.109 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.109 summary so far suite 2i.indexscans_2i.SecondaryIndexingScanTests , pass 0 , fail 1 failures so far... 2i.indexscans_2i.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index testrunner logs, diags and results are available under /data/workspace/cen006-2i-vset01-00-2i-sanity_extended/logs/testrunner-15-Jun-21_22-57-02/test_1 Run after suite setup for 2i.indexscans_2i.SecondaryIndexingScanTests.test_multi_create_query_explain_drop_index Thread Thread-3 was not properly terminated, will be terminated now. [cen006-2i-vset01-00-2i-sanity_extended] $ /bin/sh -xe /tmp/hudson1660045576181004921.sh + bash -c 'python scripts/grabdiags.py -i /tmp/tuqvm.ini;exit 0;' downloading 172.23.106.107 ...................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.108 ..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.110 ....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.111 .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.112 ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................downloading 172.23.106.109 ...........................................................................................................................................................................................................................................grabbing diags from http://172.23.106.107:8091/diag downloaded and zipped diags @ : 172.23.106.107-8091-diag.txt.gz grabbing diags from http://172.23.106.108:8091/diag downloaded and zipped diags @ : 172.23.106.108-8091-diag.txt.gz grabbing diags from http://172.23.106.110:8091/diag downloaded and zipped diags @ : 172.23.106.110-8091-diag.txt.gz grabbing diags from http://172.23.106.111:8091/diag downloaded and zipped diags @ : 172.23.106.111-8091-diag.txt.gz grabbing diags from http://172.23.106.112:8091/diag downloaded and zipped diags @ : 172.23.106.112-8091-diag.txt.gz grabbing diags from http://172.23.106.109:8091/diag downloaded and zipped diags @ : 172.23.106.109-8091-diag.txt.gz + bash -c 'python scripts/getchanges.py 4.0.0-3232;exit 0;' Traceback (most recent call last): File "scripts/getchanges.py", line 12, in builds, changes = BuildQuery().get_all_builds(version=version) File "lib/builds/build_query.py", line 279, in get_all_builds version=version, timeout=timeout, direct_build_url=direct_build_url) File "lib/builds/build_query.py", line 322, in _get_and_parse_builds raise Exception('unable to connect to %s' % (build_page + index_url)) Exception: unable to connect to http://packages.northscale.com.s3.amazonaws.com/latestbuilds/index_4.0.0.html + bash -c 'python scripts/getcoredumps.py -i /tmp/tuqvm.ini;exit 0;' looking for crashes on 172.23.106.107 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.107 Waiting for another 15 seconds (time-out after 20 min) collect core dumps info done looking for crashes on 172.23.106.108 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.108 Waiting for another 15 seconds (time-out after 20 min) collect core dumps info done looking for crashes on 172.23.106.110 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.110 Waiting for another 15 seconds (time-out after 20 min) collect core dumps info done looking for crashes on 172.23.106.111 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.111 Waiting for another 15 seconds (time-out after 20 min) collect core dumps info done looking for crashes on 172.23.106.112 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.112 Waiting for another 15 seconds (time-out after 20 min) collect core dumps info done looking for crashes on 172.23.106.109 ... erl_crash files under /opt/couchbase/var/lib/couchbase/ core* files under /opt/couchbase/var/lib/couchbase/ core* files under /tmp/ crashes not found on 172.23.106.109 Waiting for another 15 seconds (time-out after 20 min) collect core dumps info done Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 4.0.0-3232 Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Sending email to: ritam@couchbase.com prathibha@couchbase.com [BFA] Scanning build for known causes... [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE