[2016-02-11 14:19:57,828] - [rest_client] [140167256749824] - INFO - Node versions in cluster [u'4.5.0-1425-enterprise'] [2016-02-11 14:19:57,828] - [basetestcase] [140167256749824] - INFO - ============== basetestcase setup was started for test #3 bucketflush============== [2016-02-11 14:19:57,858] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.113 [2016-02-11 14:19:57,873] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.153 [2016-02-11 14:19:57,880] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.194 [2016-02-11 14:19:57,891] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.38 [2016-02-11 14:19:57,917] - [basetestcase] [140167256749824] - INFO - sleep for 10 secs. ... [2016-02-11 14:20:07,936] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.113:8091 [2016-02-11 14:20:07,943] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.113:8091 is running [2016-02-11 14:20:07,954] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.153:8091 [2016-02-11 14:20:07,962] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.153:8091 is running [2016-02-11 14:20:07,968] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.194:8091 [2016-02-11 14:20:07,974] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.194:8091 is running [2016-02-11 14:20:07,979] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.38:8091 [2016-02-11 14:20:07,984] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.38:8091 is running [2016-02-11 14:20:07,984] - [basetestcase] [140167256749824] - INFO - initializing cluster [2016-02-11 14:20:08,841] - [rest_client] [140167239964416] - INFO - settings/web params on 172.23.107.113:8091:username=Administrator&password=password&port=8091 [2016-02-11 14:20:08,864] - [rest_client] [140167239964416] - INFO - pools/default params : memoryQuota=2069 [2016-02-11 14:20:08,881] - [rest_client] [140167239964416] - INFO - settings/web params on 172.23.107.153:8091:username=Administrator&password=password&port=8091 [2016-02-11 14:20:08,906] - [rest_client] [140167239964416] - INFO - pools/default params : memoryQuota=2031 [2016-02-11 14:20:08,930] - [rest_client] [140167239964416] - INFO - settings/web params on 172.23.107.194:8091:username=Administrator&password=password&port=8091 [2016-02-11 14:20:08,956] - [rest_client] [140167239964416] - INFO - pools/default params : memoryQuota=2069 [2016-02-11 14:20:08,973] - [rest_client] [140167239964416] - INFO - settings/web params on 172.23.107.38:8091:username=Administrator&password=password&port=8091 [2016-02-11 14:20:08,990] - [rest_client] [140167239964416] - INFO - pools/default params : memoryQuota=2069 [2016-02-11 14:20:08,997] - [basetestcase] [140167256749824] - INFO - done initializing cluster [2016-02-11 14:20:10,024] - [rest_client] [140167239964416] - INFO - http://172.23.107.113:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=689&proxyPort=11211&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&replicaIndex=1&saslPassword=None [2016-02-11 14:20:10,037] - [rest_client] [140167239964416] - INFO - 0.01 seconds to create bucket default [2016-02-11 14:20:10,038] - [bucket_helper] [140167239964416] - INFO - waiting for memcached bucket : default in 172.23.107.113 to accept set ops [2016-02-11 14:20:11,244] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:20:11,389] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:20:12,734] - [task] [140167239964416] - INFO - bucket 'default' was created with per node RAM quota: 689 [2016-02-11 14:20:13,760] - [rest_client] [140167239964416] - INFO - http://172.23.107.113:8091/pools/default/buckets with param: proxyPort=11217&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket0&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=689 [2016-02-11 14:20:13,777] - [rest_client] [140167239964416] - INFO - 0.02 seconds to create bucket standard_bucket0 [2016-02-11 14:20:13,816] - [rest_client] [140167239964416] - INFO - http://172.23.107.113:8091/pools/default/buckets with param: proxyPort=11218&bucketType=membase&authType=none&replicaIndex=1&name=standard_bucket1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&replicaNumber=1&ramQuotaMB=689 [2016-02-11 14:20:13,832] - [rest_client] [140167239964416] - INFO - 0.02 seconds to create bucket standard_bucket1 [2016-02-11 14:20:13,833] - [bucket_helper] [140167239964416] - INFO - waiting for memcached bucket : standard_bucket0 in 172.23.107.113 to accept set ops [2016-02-11 14:20:18,119] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:20:18,258] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:20:19,588] - [task] [140167239964416] - INFO - bucket 'standard_bucket0' was created with per node RAM quota: 689 [2016-02-11 14:20:19,589] - [bucket_helper] [140167239964416] - INFO - waiting for memcached bucket : standard_bucket1 in 172.23.107.113 to accept set ops [2016-02-11 14:20:19,713] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:20:19,849] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:20:20,437] - [task] [140167239964416] - INFO - bucket 'standard_bucket1' was created with per node RAM quota: 689 [2016-02-11 14:20:20,481] - [basetestcase] [140167256749824] - INFO - ============== basetestcase setup was finished for test #3 bucketflush ============== [2016-02-11 14:20:20,496] - [basetestcase] [140167256749824] - INFO - sleep for 10 secs. ... [2016-02-11 14:20:31,450] - [task] [140167239964416] - INFO - adding node 172.23.107.153:8091 to cluster [2016-02-11 14:20:31,451] - [rest_client] [140167239964416] - INFO - adding remote node @172.23.107.153:8091 to this cluster @172.23.107.113:8091 [2016-02-11 14:20:36,732] - [task] [140167239964416] - INFO - adding node 172.23.107.194:8091 to cluster [2016-02-11 14:20:36,732] - [rest_client] [140167239964416] - INFO - adding remote node @172.23.107.194:8091 to this cluster @172.23.107.113:8091 [2016-02-11 14:20:38,883] - [rest_client] [140167239964416] - INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%40172.23.107.194%2Cns_1%40172.23.107.153%2Cns_1%40172.23.107.113 [2016-02-11 14:20:38,894] - [rest_client] [140167239964416] - INFO - rebalance operation started [2016-02-11 14:20:38,903] - [rest_client] [140167239964416] - INFO - rebalance percentage : 0.00 % [2016-02-11 14:20:48,925] - [rest_client] [140167239964416] - INFO - rebalance percentage : 6.84 % [2016-02-11 14:20:58,943] - [rest_client] [140167239964416] - INFO - rebalance percentage : 15.15 % [2016-02-11 14:21:08,982] - [rest_client] [140167239964416] - INFO - rebalance percentage : 23.01 % [2016-02-11 14:21:19,003] - [rest_client] [140167239964416] - INFO - rebalance percentage : 29.82 % [2016-02-11 14:21:29,028] - [rest_client] [140167239964416] - INFO - rebalance percentage : 33.74 % [2016-02-11 14:21:39,047] - [rest_client] [140167239964416] - INFO - rebalance percentage : 41.72 % [2016-02-11 14:21:49,065] - [rest_client] [140167239964416] - INFO - rebalance percentage : 50.15 % [2016-02-11 14:21:59,084] - [rest_client] [140167239964416] - INFO - rebalance percentage : 58.46 % [2016-02-11 14:22:09,102] - [rest_client] [140167239964416] - INFO - rebalance percentage : 64.18 % [2016-02-11 14:22:19,120] - [rest_client] [140167239964416] - INFO - rebalance percentage : 68.70 % [2016-02-11 14:22:29,139] - [rest_client] [140167239964416] - INFO - rebalance percentage : 76.24 % [2016-02-11 14:22:39,158] - [rest_client] [140167239964416] - INFO - rebalance percentage : 83.93 % [2016-02-11 14:22:49,177] - [rest_client] [140167239964416] - INFO - rebalance percentage : 91.99 % [2016-02-11 14:22:59,195] - [rest_client] [140167239964416] - INFO - rebalance percentage : 97.46 % [2016-02-11 14:23:09,212] - [task] [140167239964416] - INFO - rebalancing was completed with progress: 100% in 150.318374157 sec [2016-02-11 14:23:09,303] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:23:09,429] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:23:09,543] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:23:09,719] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:23:09,818] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:23:09,926] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:23:10,099] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:23:10,373] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:23:10,642] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:23:40,989] - [rest_client] [140167256749824] - INFO - Node versions in cluster [u'4.5.0-1425-enterprise', u'4.5.0-1425-enterprise', u'4.5.0-1425-enterprise'] [2016-02-11 14:23:41,007] - [basetestcase] [140167256749824] - INFO - list of all nodes in cluster: [ip:172.23.107.194 port:8091 ssh_username:root, ip:172.23.107.153 port:8091 ssh_username:root, ip:172.23.107.113 port:8091 ssh_username:root] [2016-02-11 14:23:41,359] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:23:41,495] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 243 == 0 expected on '172.23.107.113:8091', default bucket [2016-02-11 14:23:41,529] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:23:41,640] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 862 == 0 expected on '172.23.107.113:8091', standard_bucket0 bucket [2016-02-11 14:23:41,657] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:23:41,766] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 989 == 0 expected on '172.23.107.113:8091', standard_bucket1 bucket [2016-02-11 14:23:41,791] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:23:41,923] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 10459 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:23:41,945] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:23:42,070] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 9774 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:23:42,090] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:23:42,195] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 8072 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:23:42,209] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:23:42,286] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.194:8091',default bucket [2016-02-11 14:23:42,303] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:23:42,378] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.194:8091',standard_bucket0 bucket [2016-02-11 14:23:42,394] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:23:42,487] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.194:8091',standard_bucket1 bucket [2016-02-11 14:23:47,507] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.113:8091',default bucket [2016-02-11 14:23:47,522] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.113:8091',standard_bucket0 bucket [2016-02-11 14:23:47,535] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.113:8091',standard_bucket1 bucket [2016-02-11 14:23:47,549] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 9594 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:23:47,563] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 8708 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:23:47,575] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 6553 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:23:52,593] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 8905 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:23:52,603] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 7780 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:23:52,615] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 5228 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:23:57,631] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 8241 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:23:57,641] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 7014 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:23:57,652] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 4041 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:02,672] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 7401 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:02,685] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 5839 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:02,696] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 2656 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:07,713] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 6689 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:07,723] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 4916 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:07,733] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 1690 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:12,750] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 5802 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:12,764] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 3899 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:12,777] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 838 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:17,799] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 5311 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:17,811] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 3288 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:17,826] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 478 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:22,843] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 4723 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:22,854] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 2677 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:22,865] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 186 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:27,884] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 4011 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:27,899] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 2027 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:27,917] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 22 == 0 expected on '172.23.107.153:8091', standard_bucket1 bucket [2016-02-11 14:24:32,939] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 2926 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:32,952] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 1185 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:32,966] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.153:8091',standard_bucket1 bucket [2016-02-11 14:24:37,987] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 1108 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:37,999] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 247 == 0 expected on '172.23.107.153:8091', standard_bucket0 bucket [2016-02-11 14:24:43,016] - [task] [140167239964416] - WARNING - Not Ready: ep_queue_size 24 == 0 expected on '172.23.107.153:8091', default bucket [2016-02-11 14:24:43,029] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.153:8091',standard_bucket0 bucket [2016-02-11 14:24:48,046] - [task] [140167239964416] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.107.153:8091',default bucket [2016-02-11 14:24:48,173] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:24:48,309] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:24:48,439] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:24:48,578] - [task] [140167256749824] - INFO - 20000 items will be verified on default bucket [2016-02-11 14:24:48,677] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:24:48,811] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:24:48,941] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:24:49,051] - [task] [140167256749824] - INFO - 20000 items will be verified on standard_bucket0 bucket [2016-02-11 14:24:49,052] - [task] [140167228782336] - INFO - 0 items were verified [2016-02-11 14:24:49,053] - [task] [140167220389632] - INFO - 0 items were verified [2016-02-11 14:24:49,190] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:24:49,389] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:24:49,591] - [data_helper] [140167256749824] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:24:49,800] - [task] [140167256749824] - INFO - 20000 items will be verified on standard_bucket1 bucket [2016-02-11 14:24:50,055] - [task] [140167248357120] - INFO - 0 items were verified [2016-02-11 14:25:21,344] - [task] [140167220389632] - INFO - 20000 items were verified in 32.292689085 sec.the average number of ops - 619.335192745 per second [2016-02-11 14:25:21,479] - [task] [140167228782336] - INFO - 20000 items were verified in 32.9010100365 sec.the average number of ops - 607.884038161 per second [2016-02-11 14:25:21,985] - [task] [140167248357120] - INFO - 20000 items were verified in 32.1837871075 sec.the average number of ops - 621.430851357 per second [2016-02-11 14:25:22,035] - [rest_client] [140167256749824] - INFO - Node versions in cluster [u'4.5.0-1425-enterprise', u'4.5.0-1425-enterprise', u'4.5.0-1425-enterprise'] [2016-02-11 14:25:22,063] - [basetestcase] [140167256749824] - INFO - list of all nodes in cluster: [ip:172.23.107.194 port:8091 ssh_username:root, ip:172.23.107.153 port:8091 ssh_username:root, ip:172.23.107.113 port:8091 ssh_username:root] [2016-02-11 14:25:22,144] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:25:22,279] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:25:22,426] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:25:22,516] - [task] [140167239964416] - INFO - Saw curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',default bucket [2016-02-11 14:25:22,532] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:25:22,655] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:25:22,781] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:25:22,859] - [task] [140167239964416] - INFO - Saw vb_active_curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',default bucket [2016-02-11 14:25:22,873] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:25:22,991] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:25:23,115] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:25:23,182] - [task] [140167239964416] - INFO - Saw vb_replica_curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',default bucket [2016-02-11 14:25:23,206] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 default [2016-02-11 14:25:23,335] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 default [2016-02-11 14:25:23,498] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 default [2016-02-11 14:25:23,577] - [task] [140167239964416] - INFO - Saw curr_items_tot 40000 == 40000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',default bucket [2016-02-11 14:25:23,593] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:25:23,706] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:25:23,824] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:25:23,897] - [task] [140167239964416] - INFO - Saw curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket0 bucket [2016-02-11 14:25:23,926] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:25:24,061] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:25:24,190] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:25:24,260] - [task] [140167239964416] - INFO - Saw vb_active_curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket0 bucket [2016-02-11 14:25:24,279] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:25:24,394] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:25:24,552] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:25:24,615] - [task] [140167239964416] - INFO - Saw vb_replica_curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket0 bucket [2016-02-11 14:25:24,634] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket0 [2016-02-11 14:25:24,756] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket0 [2016-02-11 14:25:24,879] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket0 [2016-02-11 14:25:24,959] - [task] [140167239964416] - INFO - Saw curr_items_tot 40000 == 40000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket0 bucket [2016-02-11 14:25:24,981] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:25:25,102] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:25:25,222] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:25:25,290] - [task] [140167239964416] - INFO - Saw curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket1 bucket [2016-02-11 14:25:25,308] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:25:25,426] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:25:25,576] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:25:25,657] - [task] [140167239964416] - INFO - Saw vb_active_curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket1 bucket [2016-02-11 14:25:25,674] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:25:25,800] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:25:25,909] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:25:25,996] - [task] [140167239964416] - INFO - Saw vb_replica_curr_items 20000 == 20000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket1 bucket [2016-02-11 14:25:26,013] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.113:11210 standard_bucket1 [2016-02-11 14:25:26,141] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.153:11210 standard_bucket1 [2016-02-11 14:25:26,262] - [data_helper] [140167239964416] - INFO - creating direct client 172.23.107.194:11210 standard_bucket1 [2016-02-11 14:25:26,343] - [task] [140167239964416] - INFO - Saw curr_items_tot 40000 == 40000 expected on '172.23.107.113:8091''172.23.107.153:8091''172.23.107.194:8091',standard_bucket1 bucket [2016-02-11 14:26:21,786] - [rest_client] [140167239964416] - INFO - Flush for bucket 'default' was triggered [2016-02-11 14:27:22,855] - [rest_client] [140167239964416] - ERROR - http://172.23.107.113:8091/pools/default/buckets/standard_bucket0/controller/doFlush error 503 reason: unknown {"_":"Unexpected error: {error,{flush_wait_failed,['ns_1@172.23.107.153'],[]}}"} [2016-02-11 14:27:22,862] - [basetestcase] [140167256749824] - INFO - ============== basetestcase cleanup was started for test #3 bucketflush ============== [2016-02-11 14:27:22,989] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [u'default', u'standard_bucket0', u'standard_bucket1'] on 172.23.107.113 [2016-02-11 14:27:22,989] - [bucket_helper] [140167256749824] - INFO - remove bucket default ... [2016-02-11 14:27:25,482] - [bucket_helper] [140167256749824] - INFO - deleted bucket : default from 172.23.107.113 [2016-02-11 14:27:25,482] - [bucket_helper] [140167256749824] - INFO - waiting for bucket deletion to complete.... [2016-02-11 14:27:25,527] - [rest_client] [140167256749824] - INFO - node 172.23.107.113 existing buckets : [u'standard_bucket0', u'standard_bucket1'] [2016-02-11 14:27:25,528] - [bucket_helper] [140167256749824] - INFO - remove bucket standard_bucket0 ... [2016-02-11 14:27:26,912] - [bucket_helper] [140167256749824] - INFO - deleted bucket : standard_bucket0 from 172.23.107.113 [2016-02-11 14:27:26,912] - [bucket_helper] [140167256749824] - INFO - waiting for bucket deletion to complete.... [2016-02-11 14:27:26,936] - [rest_client] [140167256749824] - INFO - node 172.23.107.113 existing buckets : [u'standard_bucket1'] [2016-02-11 14:27:26,937] - [bucket_helper] [140167256749824] - INFO - remove bucket standard_bucket1 ... [2016-02-11 14:27:28,973] - [bucket_helper] [140167256749824] - INFO - deleted bucket : standard_bucket1 from 172.23.107.113 [2016-02-11 14:27:28,974] - [bucket_helper] [140167256749824] - INFO - waiting for bucket deletion to complete.... [2016-02-11 14:27:28,979] - [rest_client] [140167256749824] - INFO - node 172.23.107.113 existing buckets : [] [2016-02-11 14:27:28,996] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.153 [2016-02-11 14:27:29,007] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.194 [2016-02-11 14:27:29,015] - [bucket_helper] [140167256749824] - INFO - deleting existing buckets [] on 172.23.107.38 [2016-02-11 14:27:29,047] - [cluster_helper] [140167256749824] - INFO - rebalancing all nodes in order to remove nodes [2016-02-11 14:27:29,052] - [rest_client] [140167256749824] - INFO - rebalance params : password=password&ejectedNodes=ns_1%40172.23.107.194%2Cns_1%40172.23.107.153&user=Administrator&knownNodes=ns_1%40172.23.107.194%2Cns_1%40172.23.107.153%2Cns_1%40172.23.107.113 [2016-02-11 14:27:29,060] - [rest_client] [140167256749824] - INFO - rebalance operation started [2016-02-11 14:27:29,081] - [rest_client] [140167256749824] - INFO - rebalance percentage : 0.00 % [2016-02-11 14:27:39,104] - [rest_client] [140167256749824] - INFO - rebalance progress took 10.043503046 seconds [2016-02-11 14:27:39,105] - [rest_client] [140167256749824] - INFO - sleep for 10 seconds after rebalance... [2016-02-11 14:27:49,138] - [cluster_helper] [140167256749824] - INFO - removed all the nodes from cluster associated with ip:172.23.107.113 port:8091 ssh_username:root ? [(u'ns_1@172.23.107.194', 8091), (u'ns_1@172.23.107.153', 8091)] [2016-02-11 14:27:49,150] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.113:8091 [2016-02-11 14:27:49,159] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.113:8091 is running [2016-02-11 14:27:49,166] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.153:8091 [2016-02-11 14:27:49,173] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.153:8091 is running [2016-02-11 14:27:49,177] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.194:8091 [2016-02-11 14:27:49,181] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.194:8091 is running [2016-02-11 14:27:49,186] - [cluster_helper] [140167256749824] - INFO - waiting for ns_server @ 172.23.107.38:8091 [2016-02-11 14:27:49,192] - [cluster_helper] [140167256749824] - INFO - ns_server @ 172.23.107.38:8091 is running [2016-02-11 14:27:49,193] - [basetestcase] [140167256749824] - INFO - ============== basetestcase cleanup was finished for test #3 bucketflush ============== [2016-02-11 14:28:50,556] - [remote_util] [140167490725696] - INFO - connecting to 172.23.107.113 with username : root password : couchbase ssh_key: [2016-02-11 14:28:50,761] - [remote_util] [140167490725696] - INFO - Connected to 172.23.107.113 [2016-02-11 14:28:51,328] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.113: /opt/couchbase/bin/cbcollect_info 172.23.107.113-20160211-1428-diag.zip [2016-02-11 14:30:38,825] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:30:38,826] - [remote_util] [140167490725696] - INFO - uname (uname -a) - OK [2016-02-11 14:30:38,826] - [remote_util] [140167490725696] - INFO - time and TZ (date; date -u) - OK [2016-02-11 14:30:38,826] - [remote_util] [140167490725696] - INFO - ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - Exit code 127 [2016-02-11 14:30:38,826] - [remote_util] [140167490725696] - INFO - ntp peers (ntpq -p) - Exit code 127 [2016-02-11 14:30:38,826] - [remote_util] [140167490725696] - INFO - raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK [2016-02-11 14:30:38,826] - [remote_util] [140167490725696] - INFO - raw /etc/timezone (cat /etc/timezone) - Exit code 1 [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - System Hardware (lshw -json || lshw) - Exit code 127 [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - Raw /proc/vmstat (cat /proc/vmstat) - OK [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - Raw /proc/mounts (cat /proc/mounts) - OK [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - Raw /proc/partitions (cat /proc/partitions) - OK [2016-02-11 14:30:38,827] - [remote_util] [140167490725696] - INFO - Raw /proc/diskstats (cat /proc/diskstats; echo '') - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 1.000000 seconds - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 3 after 1.000000 seconds - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 4 after 1.000000 seconds - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 5 after 1.000000 seconds - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 6 after 1.000000 seconds - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 7 after 1.000000 seconds - OK [2016-02-11 14:30:38,828] - [remote_util] [140167490725696] - INFO - Taking sample 8 after 1.000000 seconds - OK [2016-02-11 14:30:38,829] - [remote_util] [140167490725696] - INFO - Taking sample 9 after 1.000000 seconds - OK [2016-02-11 14:30:38,829] - [remote_util] [140167490725696] - INFO - Taking sample 10 after 1.000000 seconds - OK [2016-02-11 14:30:38,829] - [remote_util] [140167490725696] - INFO - Raw /proc/interrupts (cat /proc/interrupts) - OK [2016-02-11 14:30:38,829] - [remote_util] [140167490725696] - INFO - Swap configuration (free -t) - OK [2016-02-11 14:30:38,829] - [remote_util] [140167490725696] - INFO - Swap configuration (swapon -s) - OK [2016-02-11 14:30:38,829] - [remote_util] [140167490725696] - INFO - Kernel modules (lsmod) - OK [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/redhat-release) - OK [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Distro version (lsb_release -a) - OK [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/SuSE-release) - Exit code 1 [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/issue) - OK [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Installed software (rpm -qa) - OK [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Installed software (COLUMNS=300 dpkg -l) - Exit code 127 [2016-02-11 14:30:38,830] - [remote_util] [140167490725696] - INFO - Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - sysctl settings (sysctl -a) - OK [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - Relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - LVM info (lvdisplay) - OK [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - LVM info (vgdisplay) - OK [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - LVM info (pvdisplay) - OK [2016-02-11 14:30:38,831] - [remote_util] [140167490725696] - INFO - Network configuration (ifconfig -a) - OK [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 10.000000 seconds - OK [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Raw /proc/net/dev (cat /proc/net/dev) - OK [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Network link statistics (ip -s link) - OK [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Network status (netstat -anp || netstat -an) - OK [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Network routing table (netstat -rn) - OK [2016-02-11 14:30:38,832] - [remote_util] [140167490725696] - INFO - Network socket statistics (ss -an) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Extended socket statistics (ss -an --info --processes) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Arp cache (arp -na) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Iptables dump (iptables-save) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Raw /etc/hosts (cat /etc/hosts) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK [2016-02-11 14:30:38,833] - [remote_util] [140167490725696] - INFO - Filesystem (df -ha) - OK [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - System activity reporter (sar 1 10) - Exit code 127 [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - System paging activity (vmstat 1 10) - OK [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - System uptime (uptime) - OK [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - couchbase user definition (getent passwd couchbase) - OK [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:30:38,834] - [remote_util] [140167490725696] - INFO - Interrupt status (intrstat 1 10) - Exit code 127 [2016-02-11 14:30:38,835] - [remote_util] [140167490725696] - INFO - Processor status (mpstat 1 10) - Exit code 127 [2016-02-11 14:30:38,835] - [remote_util] [140167490725696] - INFO - System log (cat /var/adm/messages) - Exit code 1 [2016-02-11 14:30:38,835] - [remote_util] [140167490725696] - INFO - Raw /proc/uptime (cat /proc/uptime) - OK [2016-02-11 14:30:38,835] - [remote_util] [140167490725696] - INFO - All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 [2016-02-11 14:30:38,835] - [remote_util] [140167490725696] - INFO - Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK [2016-02-11 14:30:38,836] - [remote_util] [140167490725696] - INFO - Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK [2016-02-11 14:30:38,836] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --hardware) - Exit code 127 [2016-02-11 14:30:38,836] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --show) - Exit code 127 [2016-02-11 14:30:38,836] - [remote_util] [140167490725696] - INFO - NUMA data (cat /sys/devices/system/node/node*/numastat) - OK [2016-02-11 14:30:38,836] - [remote_util] [140167490725696] - INFO - Kernel log buffer (dmesg -H || dmesg) - OK [2016-02-11 14:30:38,837] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:30:38,837] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:30:38,837] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:30:38,838] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:30:38,838] - [remote_util] [140167490725696] - INFO - Network statistics (netstat -s) - OK [2016-02-11 14:30:38,838] - [remote_util] [140167490725696] - INFO - Full raw netstat (cat /proc/net/netstat) - OK [2016-02-11 14:30:38,838] - [remote_util] [140167490725696] - INFO - CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 [2016-02-11 14:30:38,839] - [remote_util] [140167490725696] - INFO - Found dump-guts: /opt/couchbase/bin/dump-guts [2016-02-11 14:30:38,839] - [remote_util] [140167490725696] - INFO - Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... [2016-02-11 14:30:38,839] - [remote_util] [140167490725696] - INFO - product diag header () - OK [2016-02-11 14:30:38,840] - [remote_util] [140167490725696] - INFO - Found escript-wrapper: /opt/couchbase/bin/escript-wrapper [2016-02-11 14:30:38,840] - [remote_util] [140167490725696] - INFO - Found dump-stats: /opt/couchbase/bin/dump-stats [2016-02-11 14:30:38,840] - [remote_util] [140167490725696] - INFO - Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK [2016-02-11 14:30:38,840] - [remote_util] [140167490725696] - INFO - Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:30:38,841] - [remote_util] [140167490725696] - INFO - Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:30:38,841] - [remote_util] [140167490725696] - INFO - couch_dbinfo (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-name', '*.couch.*', '-exec', 'couch_dbinfo', '{}', '+']) - OK [2016-02-11 14:30:38,841] - [remote_util] [140167490725696] - INFO - Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:30:38,842] - [remote_util] [140167490725696] - INFO - Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:30:38,842] - [remote_util] [140167490725696] - INFO - Version file (cat '/opt/couchbase/VERSION.txt') - OK [2016-02-11 14:30:38,842] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.txt') - Exit code 1 [2016-02-11 14:30:38,842] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.xml') - OK [2016-02-11 14:30:38,842] - [remote_util] [140167490725696] - INFO - Couchbase config () - OK [2016-02-11 14:30:38,843] - [remote_util] [140167490725696] - INFO - Couchbase static config () - OK [2016-02-11 14:30:38,843] - [remote_util] [140167490725696] - INFO - Raw ns_log () - OK [2016-02-11 14:30:38,843] - [remote_util] [140167490725696] - INFO - Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK [2016-02-11 14:30:38,843] - [remote_util] [140167490725696] - INFO - Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK [2016-02-11 14:30:38,844] - [remote_util] [140167490725696] - INFO - couchbase diags (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag?noLogs=1']) - OK [2016-02-11 14:30:38,844] - [remote_util] [140167490725696] - INFO - master events (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/masterEvents?o=1']) - OK [2016-02-11 14:30:38,844] - [remote_util] [140167490725696] - INFO - ale configuration (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/ale']) - OK [2016-02-11 14:30:38,844] - [remote_util] [140167490725696] - INFO - couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK [2016-02-11 14:30:38,845] - [remote_util] [140167490725696] - INFO - couchbase logs (info.log) (cbbrowse_logs info.log) - OK [2016-02-11 14:30:38,845] - [remote_util] [140167490725696] - INFO - couchbase logs (error.log) (cbbrowse_logs error.log) - OK [2016-02-11 14:30:38,845] - [remote_util] [140167490725696] - INFO - couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK [2016-02-11 14:30:38,845] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK [2016-02-11 14:30:38,846] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK [2016-02-11 14:30:38,846] - [remote_util] [140167490725696] - INFO - couchbase logs (views.log) (cbbrowse_logs views.log) - OK [2016-02-11 14:30:38,846] - [remote_util] [140167490725696] - INFO - couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK [2016-02-11 14:30:38,846] - [remote_util] [140167490725696] - INFO - couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK [2016-02-11 14:30:38,847] - [remote_util] [140167490725696] - INFO - couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK [2016-02-11 14:30:38,847] - [remote_util] [140167490725696] - INFO - couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK [2016-02-11 14:30:38,847] - [remote_util] [140167490725696] - INFO - couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK [2016-02-11 14:30:38,847] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK [2016-02-11 14:30:38,847] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK [2016-02-11 14:30:38,848] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK [2016-02-11 14:30:38,848] - [remote_util] [140167490725696] - INFO - couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK [2016-02-11 14:30:38,848] - [remote_util] [140167490725696] - INFO - couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK [2016-02-11 14:30:38,848] - [remote_util] [140167490725696] - INFO - couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 [2016-02-11 14:30:38,849] - [remote_util] [140167490725696] - INFO - couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK [2016-02-11 14:30:38,849] - [remote_util] [140167490725696] - INFO - couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 [2016-02-11 14:30:38,849] - [remote_util] [140167490725696] - INFO - couchbase logs (fts.log) (cbbrowse_logs fts.log) - Exit code 1 [2016-02-11 14:30:38,849] - [remote_util] [140167490725696] - INFO - couchbase logs (metakv.log) (cbbrowse_logs metakv.log) - OK [2016-02-11 14:30:38,850] - [remote_util] [140167490725696] - INFO - memcached stats all (['cbstats', '-a', '127.0.0.1:11209', 'all', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,850] - [remote_util] [140167490725696] - INFO - memcached stats allocator (['cbstats', '-a', '127.0.0.1:11209', 'allocator', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,850] - [remote_util] [140167490725696] - INFO - memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11209', 'checkpoint', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,850] - [remote_util] [140167490725696] - INFO - memcached stats config (['cbstats', '-a', '127.0.0.1:11209', 'config', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats dcp (['cbstats', '-a', '127.0.0.1:11209', 'dcp', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11209', 'dcpagg', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats ['diskinfo', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'diskinfo', 'detail', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11209', 'dispatcher', 'logs', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats failovers (['cbstats', '-a', '127.0.0.1:11209', 'failovers', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats ['hash', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'hash', 'detail', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,851] - [remote_util] [140167490725696] - INFO - memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11209', 'kvstore', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,852] - [remote_util] [140167490725696] - INFO - memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11209', 'kvtimings', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,852] - [remote_util] [140167490725696] - INFO - memcached stats memory (['cbstats', '-a', '127.0.0.1:11209', 'memory', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,852] - [remote_util] [140167490725696] - INFO - memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11209', 'prev-vbucket', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,852] - [remote_util] [140167490725696] - INFO - memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11209', 'runtimes', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,852] - [remote_util] [140167490725696] - INFO - memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11209', 'scheduler', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,852] - [remote_util] [140167490725696] - INFO - memcached stats tap (['cbstats', '-a', '127.0.0.1:11209', 'tap', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11209', 'tapagg', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats timings (['cbstats', '-a', '127.0.0.1:11209', 'timings', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats uuid (['cbstats', '-a', '127.0.0.1:11209', 'uuid', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11209', 'vbucket', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-details', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-seqno (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-seqno', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,853] - [remote_util] [140167490725696] - INFO - memcached stats warmup (['cbstats', '-a', '127.0.0.1:11209', 'warmup', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - memcached stats workload (['cbstats', '-a', '127.0.0.1:11209', 'workload', '-b', '_admin', '-p', '61e36e37720a0948f7d316ef000722ec']) - Exit code 1 [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11209', '-u', '_admin', '-P', '61e36e37720a0948f7d316ef000722ec', 'connections']) - OK [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', '61e36e37720a0948f7d316ef000722ec', '-h', '127.0.0.1:11209', '-v']) - OK [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', '61e36e37720a0948f7d316ef000722ec', '-h', '127.0.0.1:11209', '-v', 'subdoc_execute']) - OK [2016-02-11 14:30:38,854] - [remote_util] [140167490725696] - INFO - stats archives (['escript', '/opt/couchbase/bin/escript-wrapper', '--initargs-path', '/opt/couchbase/var/lib/couchbase/initargs', '--', '/opt/couchbase/bin/dump-stats', '/opt/couchbase/var/lib/couchbase/stats']) - OK [2016-02-11 14:30:38,855] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/couchbase.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/couchbase.log [2016-02-11 14:30:38,855] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ddocs.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ddocs.log [2016-02-11 14:30:38,855] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.error.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.error.log [2016-02-11 14:30:38,855] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.ns_couchdb.log [2016-02-11 14:30:38,855] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/stats.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/stats.log [2016-02-11 14:30:38,855] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/memcached.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/memcached.log [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.ssl_proxy.log [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.indexer.log [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.xdcr_errors.log [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.fts.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.fts.log [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.http_access_internal.log [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/syslog.tar.gz -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/syslog.tar.gz [2016-02-11 14:30:38,856] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.reports.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.reports.log [2016-02-11 14:30:38,857] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.query.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.query.log [2016-02-11 14:30:38,857] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.couchdb.log [2016-02-11 14:30:38,857] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.goxdcr.log [2016-02-11 14:30:38,857] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/master_events.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/master_events.log [2016-02-11 14:30:38,857] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.info.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.info.log [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/diag.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/diag.log [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.stats.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.stats.log [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.xdcr.log [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.views.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.views.log [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ini.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ini.log [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/stats_archives.json -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/stats_archives.json [2016-02-11 14:30:38,858] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.mapreduce_errors.log [2016-02-11 14:30:38,859] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.projector.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.projector.log [2016-02-11 14:30:38,859] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.metakv.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.metakv.log [2016-02-11 14:30:38,859] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.xdcr_trace.log [2016-02-11 14:30:38,859] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.debug.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.debug.log [2016-02-11 14:30:38,859] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.babysitter.log [2016-02-11 14:30:38,859] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpUTFA5A/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.107.113_20160211-222734/ns_server.http_access.log [2016-02-11 14:30:39,303] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.113: rm -f /root/172.23.107.113-20160211-1428-diag.zip [2016-02-11 14:30:39,348] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:30:39,351] - [remote_util] [140167490725696] - INFO - connecting to 172.23.107.153 with username : root password : couchbase ssh_key: [2016-02-11 14:30:39,555] - [remote_util] [140167490725696] - INFO - Connected to 172.23.107.153 [2016-02-11 14:30:40,172] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.153: /opt/couchbase/bin/cbcollect_info 172.23.107.153-20160211-1430-diag.zip [2016-02-11 14:31:56,350] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:31:56,351] - [remote_util] [140167490725696] - INFO - uname (uname -a) - OK [2016-02-11 14:31:56,351] - [remote_util] [140167490725696] - INFO - time and TZ (date; date -u) - OK [2016-02-11 14:31:56,351] - [remote_util] [140167490725696] - INFO - ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - Exit code 127 [2016-02-11 14:31:56,351] - [remote_util] [140167490725696] - INFO - ntp peers (ntpq -p) - Exit code 127 [2016-02-11 14:31:56,351] - [remote_util] [140167490725696] - INFO - raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK [2016-02-11 14:31:56,351] - [remote_util] [140167490725696] - INFO - raw /etc/timezone (cat /etc/timezone) - Exit code 1 [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - System Hardware (lshw -json || lshw) - Exit code 127 [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - Raw /proc/vmstat (cat /proc/vmstat) - OK [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - Raw /proc/mounts (cat /proc/mounts) - OK [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - Raw /proc/partitions (cat /proc/partitions) - OK [2016-02-11 14:31:56,352] - [remote_util] [140167490725696] - INFO - Raw /proc/diskstats (cat /proc/diskstats; echo '') - OK [2016-02-11 14:31:56,353] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 1.000000 seconds - OK [2016-02-11 14:31:56,353] - [remote_util] [140167490725696] - INFO - Taking sample 3 after 1.000000 seconds - OK [2016-02-11 14:31:56,353] - [remote_util] [140167490725696] - INFO - Taking sample 4 after 1.000000 seconds - OK [2016-02-11 14:31:56,353] - [remote_util] [140167490725696] - INFO - Taking sample 5 after 1.000000 seconds - OK [2016-02-11 14:31:56,353] - [remote_util] [140167490725696] - INFO - Taking sample 6 after 1.000000 seconds - OK [2016-02-11 14:31:56,353] - [remote_util] [140167490725696] - INFO - Taking sample 7 after 1.000000 seconds - OK [2016-02-11 14:31:56,354] - [remote_util] [140167490725696] - INFO - Taking sample 8 after 1.000000 seconds - OK [2016-02-11 14:31:56,354] - [remote_util] [140167490725696] - INFO - Taking sample 9 after 1.000000 seconds - OK [2016-02-11 14:31:56,354] - [remote_util] [140167490725696] - INFO - Taking sample 10 after 1.000000 seconds - OK [2016-02-11 14:31:56,354] - [remote_util] [140167490725696] - INFO - Raw /proc/interrupts (cat /proc/interrupts) - OK [2016-02-11 14:31:56,354] - [remote_util] [140167490725696] - INFO - Swap configuration (free -t) - OK [2016-02-11 14:31:56,355] - [remote_util] [140167490725696] - INFO - Swap configuration (swapon -s) - OK [2016-02-11 14:31:56,355] - [remote_util] [140167490725696] - INFO - Kernel modules (lsmod) - OK [2016-02-11 14:31:56,355] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/redhat-release) - OK [2016-02-11 14:31:56,355] - [remote_util] [140167490725696] - INFO - Distro version (lsb_release -a) - Exit code 127 [2016-02-11 14:31:56,356] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/SuSE-release) - Exit code 1 [2016-02-11 14:31:56,356] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/issue) - OK [2016-02-11 14:31:56,356] - [remote_util] [140167490725696] - INFO - Installed software (rpm -qa) - OK [2016-02-11 14:31:56,356] - [remote_util] [140167490725696] - INFO - Installed software (COLUMNS=300 dpkg -l) - Exit code 127 [2016-02-11 14:31:56,357] - [remote_util] [140167490725696] - INFO - Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 [2016-02-11 14:31:56,357] - [remote_util] [140167490725696] - INFO - Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK [2016-02-11 14:31:56,357] - [remote_util] [140167490725696] - INFO - sysctl settings (sysctl -a) - OK [2016-02-11 14:31:56,357] - [remote_util] [140167490725696] - INFO - Relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 [2016-02-11 14:31:56,357] - [remote_util] [140167490725696] - INFO - LVM info (lvdisplay) - OK [2016-02-11 14:31:56,358] - [remote_util] [140167490725696] - INFO - LVM info (vgdisplay) - OK [2016-02-11 14:31:56,358] - [remote_util] [140167490725696] - INFO - LVM info (pvdisplay) - OK [2016-02-11 14:31:56,358] - [remote_util] [140167490725696] - INFO - Network configuration (ifconfig -a) - OK [2016-02-11 14:31:56,359] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 10.000000 seconds - OK [2016-02-11 14:31:56,359] - [remote_util] [140167490725696] - INFO - Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 [2016-02-11 14:31:56,359] - [remote_util] [140167490725696] - INFO - Raw /proc/net/dev (cat /proc/net/dev) - OK [2016-02-11 14:31:56,359] - [remote_util] [140167490725696] - INFO - Network link statistics (ip -s link) - OK [2016-02-11 14:31:56,359] - [remote_util] [140167490725696] - INFO - Network status (netstat -anp || netstat -an) - OK [2016-02-11 14:31:56,360] - [remote_util] [140167490725696] - INFO - Network routing table (netstat -rn) - OK [2016-02-11 14:31:56,360] - [remote_util] [140167490725696] - INFO - Network socket statistics (ss -an) - OK [2016-02-11 14:31:56,360] - [remote_util] [140167490725696] - INFO - Extended socket statistics (ss -an --info --processes) - OK [2016-02-11 14:31:56,361] - [remote_util] [140167490725696] - INFO - Arp cache (arp -na) - OK [2016-02-11 14:31:56,361] - [remote_util] [140167490725696] - INFO - Iptables dump (iptables-save) - OK [2016-02-11 14:31:56,361] - [remote_util] [140167490725696] - INFO - Raw /etc/hosts (cat /etc/hosts) - OK [2016-02-11 14:31:56,361] - [remote_util] [140167490725696] - INFO - Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK [2016-02-11 14:31:56,362] - [remote_util] [140167490725696] - INFO - Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK [2016-02-11 14:31:56,362] - [remote_util] [140167490725696] - INFO - Filesystem (df -ha) - OK [2016-02-11 14:31:56,362] - [remote_util] [140167490725696] - INFO - System activity reporter (sar 1 10) - Exit code 127 [2016-02-11 14:31:56,362] - [remote_util] [140167490725696] - INFO - System paging activity (vmstat 1 10) - OK [2016-02-11 14:31:56,363] - [remote_util] [140167490725696] - INFO - System uptime (uptime) - OK [2016-02-11 14:31:56,363] - [remote_util] [140167490725696] - INFO - couchbase user definition (getent passwd couchbase) - OK [2016-02-11 14:31:56,363] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:31:56,364] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:31:56,364] - [remote_util] [140167490725696] - INFO - Interrupt status (intrstat 1 10) - Exit code 127 [2016-02-11 14:31:56,364] - [remote_util] [140167490725696] - INFO - Processor status (mpstat 1 10) - Exit code 127 [2016-02-11 14:31:56,364] - [remote_util] [140167490725696] - INFO - System log (cat /var/adm/messages) - Exit code 1 [2016-02-11 14:31:56,364] - [remote_util] [140167490725696] - INFO - Raw /proc/uptime (cat /proc/uptime) - OK [2016-02-11 14:31:56,364] - [remote_util] [140167490725696] - INFO - All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --hardware) - Exit code 127 [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --show) - Exit code 127 [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - NUMA data (cat /sys/devices/system/node/node*/numastat) - OK [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - Kernel log buffer (dmesg -H || dmesg) - OK [2016-02-11 14:31:56,365] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - Network statistics (netstat -s) - OK [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - Full raw netstat (cat /proc/net/netstat) - OK [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 [2016-02-11 14:31:56,366] - [remote_util] [140167490725696] - INFO - Found dump-guts: /opt/couchbase/bin/dump-guts [2016-02-11 14:31:56,367] - [remote_util] [140167490725696] - INFO - Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... [2016-02-11 14:31:56,367] - [remote_util] [140167490725696] - INFO - product diag header () - OK [2016-02-11 14:31:56,367] - [remote_util] [140167490725696] - INFO - Found escript-wrapper: /opt/couchbase/bin/escript-wrapper [2016-02-11 14:31:56,367] - [remote_util] [140167490725696] - INFO - Found dump-stats: /opt/couchbase/bin/dump-stats [2016-02-11 14:31:56,367] - [remote_util] [140167490725696] - INFO - Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK [2016-02-11 14:31:56,367] - [remote_util] [140167490725696] - INFO - Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:31:56,368] - [remote_util] [140167490725696] - INFO - Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:31:56,368] - [remote_util] [140167490725696] - INFO - couch_dbinfo (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-name', '*.couch.*', '-exec', 'couch_dbinfo', '{}', '+']) - OK [2016-02-11 14:31:56,368] - [remote_util] [140167490725696] - INFO - Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:31:56,368] - [remote_util] [140167490725696] - INFO - Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:31:56,369] - [remote_util] [140167490725696] - INFO - Version file (cat '/opt/couchbase/VERSION.txt') - OK [2016-02-11 14:31:56,369] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.txt') - Exit code 1 [2016-02-11 14:31:56,369] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.xml') - OK [2016-02-11 14:31:56,369] - [remote_util] [140167490725696] - INFO - Couchbase config () - OK [2016-02-11 14:31:56,370] - [remote_util] [140167490725696] - INFO - Couchbase static config () - OK [2016-02-11 14:31:56,370] - [remote_util] [140167490725696] - INFO - Raw ns_log () - OK [2016-02-11 14:31:56,370] - [remote_util] [140167490725696] - INFO - Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK [2016-02-11 14:31:56,370] - [remote_util] [140167490725696] - INFO - Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK [2016-02-11 14:31:56,371] - [remote_util] [140167490725696] - INFO - couchbase diags (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag?noLogs=1']) - OK [2016-02-11 14:31:56,371] - [remote_util] [140167490725696] - INFO - master events (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/masterEvents?o=1']) - OK [2016-02-11 14:31:56,371] - [remote_util] [140167490725696] - INFO - ale configuration (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/ale']) - OK [2016-02-11 14:31:56,371] - [remote_util] [140167490725696] - INFO - couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK [2016-02-11 14:31:56,372] - [remote_util] [140167490725696] - INFO - couchbase logs (info.log) (cbbrowse_logs info.log) - OK [2016-02-11 14:31:56,372] - [remote_util] [140167490725696] - INFO - couchbase logs (error.log) (cbbrowse_logs error.log) - OK [2016-02-11 14:31:56,372] - [remote_util] [140167490725696] - INFO - couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK [2016-02-11 14:31:56,372] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK [2016-02-11 14:31:56,373] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK [2016-02-11 14:31:56,373] - [remote_util] [140167490725696] - INFO - couchbase logs (views.log) (cbbrowse_logs views.log) - OK [2016-02-11 14:31:56,373] - [remote_util] [140167490725696] - INFO - couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK [2016-02-11 14:31:56,373] - [remote_util] [140167490725696] - INFO - couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK [2016-02-11 14:31:56,373] - [remote_util] [140167490725696] - INFO - couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK [2016-02-11 14:31:56,374] - [remote_util] [140167490725696] - INFO - couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK [2016-02-11 14:31:56,374] - [remote_util] [140167490725696] - INFO - couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK [2016-02-11 14:31:56,374] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK [2016-02-11 14:31:56,374] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK [2016-02-11 14:31:56,375] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK [2016-02-11 14:31:56,375] - [remote_util] [140167490725696] - INFO - couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK [2016-02-11 14:31:56,375] - [remote_util] [140167490725696] - INFO - couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK [2016-02-11 14:31:56,375] - [remote_util] [140167490725696] - INFO - couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 [2016-02-11 14:31:56,375] - [remote_util] [140167490725696] - INFO - couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK [2016-02-11 14:31:56,376] - [remote_util] [140167490725696] - INFO - couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 [2016-02-11 14:31:56,376] - [remote_util] [140167490725696] - INFO - couchbase logs (fts.log) (cbbrowse_logs fts.log) - Exit code 1 [2016-02-11 14:31:56,376] - [remote_util] [140167490725696] - INFO - couchbase logs (metakv.log) (cbbrowse_logs metakv.log) - OK [2016-02-11 14:31:56,376] - [remote_util] [140167490725696] - INFO - memcached stats all (['cbstats', '-a', '127.0.0.1:11209', 'all', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,376] - [remote_util] [140167490725696] - INFO - memcached stats allocator (['cbstats', '-a', '127.0.0.1:11209', 'allocator', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,376] - [remote_util] [140167490725696] - INFO - memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11209', 'checkpoint', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats config (['cbstats', '-a', '127.0.0.1:11209', 'config', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats dcp (['cbstats', '-a', '127.0.0.1:11209', 'dcp', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11209', 'dcpagg', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats ['diskinfo', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'diskinfo', 'detail', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11209', 'dispatcher', 'logs', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats failovers (['cbstats', '-a', '127.0.0.1:11209', 'failovers', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,377] - [remote_util] [140167490725696] - INFO - memcached stats ['hash', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'hash', 'detail', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,378] - [remote_util] [140167490725696] - INFO - memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11209', 'kvstore', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,378] - [remote_util] [140167490725696] - INFO - memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11209', 'kvtimings', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,378] - [remote_util] [140167490725696] - INFO - memcached stats memory (['cbstats', '-a', '127.0.0.1:11209', 'memory', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,378] - [remote_util] [140167490725696] - INFO - memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11209', 'prev-vbucket', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,378] - [remote_util] [140167490725696] - INFO - memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11209', 'runtimes', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,378] - [remote_util] [140167490725696] - INFO - memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11209', 'scheduler', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats tap (['cbstats', '-a', '127.0.0.1:11209', 'tap', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11209', 'tapagg', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats timings (['cbstats', '-a', '127.0.0.1:11209', 'timings', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats uuid (['cbstats', '-a', '127.0.0.1:11209', 'uuid', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11209', 'vbucket', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-details', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,379] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-seqno (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-seqno', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - memcached stats warmup (['cbstats', '-a', '127.0.0.1:11209', 'warmup', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - memcached stats workload (['cbstats', '-a', '127.0.0.1:11209', 'workload', '-b', '_admin', '-p', '13e80d9f7132a070ce50b8887d34e2da']) - Exit code 1 [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11209', '-u', '_admin', '-P', '13e80d9f7132a070ce50b8887d34e2da', 'connections']) - OK [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', '13e80d9f7132a070ce50b8887d34e2da', '-h', '127.0.0.1:11209', '-v']) - OK [2016-02-11 14:31:56,380] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', '13e80d9f7132a070ce50b8887d34e2da', '-h', '127.0.0.1:11209', '-v', 'subdoc_execute']) - OK [2016-02-11 14:31:56,381] - [remote_util] [140167490725696] - INFO - stats archives (['escript', '/opt/couchbase/bin/escript-wrapper', '--initargs-path', '/opt/couchbase/var/lib/couchbase/initargs', '--', '/opt/couchbase/bin/dump-stats', '/opt/couchbase/var/lib/couchbase/stats']) - OK [2016-02-11 14:31:56,381] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/couchbase.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/couchbase.log [2016-02-11 14:31:56,381] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ddocs.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ddocs.log [2016-02-11 14:31:56,381] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.error.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.error.log [2016-02-11 14:31:56,381] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.ns_couchdb.log [2016-02-11 14:31:56,381] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/stats.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/stats.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/memcached.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/memcached.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.ssl_proxy.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.indexer.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.xdcr_errors.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.fts.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.fts.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.http_access_internal.log [2016-02-11 14:31:56,382] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/syslog.tar.gz -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/syslog.tar.gz [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.reports.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.reports.log [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.query.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.query.log [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.couchdb.log [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.goxdcr.log [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/master_events.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/master_events.log [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.info.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.info.log [2016-02-11 14:31:56,383] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/diag.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/diag.log [2016-02-11 14:31:56,384] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.stats.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.stats.log [2016-02-11 14:31:56,384] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.xdcr.log [2016-02-11 14:31:56,384] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.views.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.views.log [2016-02-11 14:31:56,384] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ini.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ini.log [2016-02-11 14:31:56,384] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/stats_archives.json -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/stats_archives.json [2016-02-11 14:31:56,384] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.mapreduce_errors.log [2016-02-11 14:31:56,385] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.projector.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.projector.log [2016-02-11 14:31:56,385] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.metakv.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.metakv.log [2016-02-11 14:31:56,385] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.xdcr_trace.log [2016-02-11 14:31:56,385] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.debug.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.debug.log [2016-02-11 14:31:56,385] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.babysitter.log [2016-02-11 14:31:56,385] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpVcHzGX/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.107.153_20160211-223039/ns_server.http_access.log [2016-02-11 14:31:56,920] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.153: rm -f /root/172.23.107.153-20160211-1430-diag.zip [2016-02-11 14:31:56,957] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:31:56,959] - [remote_util] [140167490725696] - INFO - connecting to 172.23.107.194 with username : root password : couchbase ssh_key: [2016-02-11 14:31:57,127] - [remote_util] [140167490725696] - INFO - Connected to 172.23.107.194 [2016-02-11 14:31:57,663] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.194: /opt/couchbase/bin/cbcollect_info 172.23.107.194-20160211-1431-diag.zip [2016-02-11 14:32:59,478] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:32:59,479] - [remote_util] [140167490725696] - INFO - uname (uname -a) - OK [2016-02-11 14:32:59,479] - [remote_util] [140167490725696] - INFO - time and TZ (date; date -u) - OK [2016-02-11 14:32:59,479] - [remote_util] [140167490725696] - INFO - ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - OK [2016-02-11 14:32:59,479] - [remote_util] [140167490725696] - INFO - ntp peers (ntpq -p) - OK [2016-02-11 14:32:59,479] - [remote_util] [140167490725696] - INFO - raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK [2016-02-11 14:32:59,480] - [remote_util] [140167490725696] - INFO - raw /etc/timezone (cat /etc/timezone) - Exit code 1 [2016-02-11 14:32:59,480] - [remote_util] [140167490725696] - INFO - System Hardware (lshw -json || lshw) - Exit code 127 [2016-02-11 14:32:59,480] - [remote_util] [140167490725696] - INFO - Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK [2016-02-11 14:32:59,480] - [remote_util] [140167490725696] - INFO - Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK [2016-02-11 14:32:59,480] - [remote_util] [140167490725696] - INFO - Raw /proc/vmstat (cat /proc/vmstat) - OK [2016-02-11 14:32:59,480] - [remote_util] [140167490725696] - INFO - Raw /proc/mounts (cat /proc/mounts) - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Raw /proc/partitions (cat /proc/partitions) - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Raw /proc/diskstats (cat /proc/diskstats; echo '') - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 1.000000 seconds - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Taking sample 3 after 1.000000 seconds - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Taking sample 4 after 1.000000 seconds - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Taking sample 5 after 1.000000 seconds - OK [2016-02-11 14:32:59,481] - [remote_util] [140167490725696] - INFO - Taking sample 6 after 1.000000 seconds - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Taking sample 7 after 1.000000 seconds - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Taking sample 8 after 1.000000 seconds - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Taking sample 9 after 1.000000 seconds - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Taking sample 10 after 1.000000 seconds - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Raw /proc/interrupts (cat /proc/interrupts) - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Swap configuration (free -t) - OK [2016-02-11 14:32:59,482] - [remote_util] [140167490725696] - INFO - Swap configuration (swapon -s) - OK [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Kernel modules (lsmod) - OK [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/redhat-release) - OK [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Distro version (lsb_release -a) - OK [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/SuSE-release) - Exit code 1 [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/issue) - OK [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Installed software (rpm -qa) - OK [2016-02-11 14:32:59,483] - [remote_util] [140167490725696] - INFO - Installed software (COLUMNS=300 dpkg -l) - Exit code 127 [2016-02-11 14:32:59,484] - [remote_util] [140167490725696] - INFO - Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 [2016-02-11 14:32:59,484] - [remote_util] [140167490725696] - INFO - Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK [2016-02-11 14:32:59,484] - [remote_util] [140167490725696] - INFO - sysctl settings (sysctl -a) - OK [2016-02-11 14:32:59,484] - [remote_util] [140167490725696] - INFO - Relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 [2016-02-11 14:32:59,484] - [remote_util] [140167490725696] - INFO - LVM info (lvdisplay) - OK [2016-02-11 14:32:59,484] - [remote_util] [140167490725696] - INFO - LVM info (vgdisplay) - OK [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - LVM info (pvdisplay) - OK [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - Network configuration (ifconfig -a) - OK [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 10.000000 seconds - OK [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - Raw /proc/net/dev (cat /proc/net/dev) - OK [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - Network link statistics (ip -s link) - OK [2016-02-11 14:32:59,485] - [remote_util] [140167490725696] - INFO - Network status (netstat -anp || netstat -an) - OK [2016-02-11 14:32:59,486] - [remote_util] [140167490725696] - INFO - Network routing table (netstat -rn) - OK [2016-02-11 14:32:59,486] - [remote_util] [140167490725696] - INFO - Network socket statistics (ss -an) - OK [2016-02-11 14:32:59,486] - [remote_util] [140167490725696] - INFO - Extended socket statistics (ss -an --info --processes) - OK [2016-02-11 14:32:59,486] - [remote_util] [140167490725696] - INFO - Arp cache (arp -na) - OK [2016-02-11 14:32:59,487] - [remote_util] [140167490725696] - INFO - Iptables dump (iptables-save) - OK [2016-02-11 14:32:59,487] - [remote_util] [140167490725696] - INFO - Raw /etc/hosts (cat /etc/hosts) - OK [2016-02-11 14:32:59,487] - [remote_util] [140167490725696] - INFO - Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK [2016-02-11 14:32:59,487] - [remote_util] [140167490725696] - INFO - Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK [2016-02-11 14:32:59,487] - [remote_util] [140167490725696] - INFO - Filesystem (df -ha) - OK [2016-02-11 14:32:59,488] - [remote_util] [140167490725696] - INFO - System activity reporter (sar 1 10) - Exit code 127 [2016-02-11 14:32:59,488] - [remote_util] [140167490725696] - INFO - System paging activity (vmstat 1 10) - OK [2016-02-11 14:32:59,488] - [remote_util] [140167490725696] - INFO - System uptime (uptime) - OK [2016-02-11 14:32:59,489] - [remote_util] [140167490725696] - INFO - couchbase user definition (getent passwd couchbase) - OK [2016-02-11 14:32:59,489] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:32:59,489] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:32:59,489] - [remote_util] [140167490725696] - INFO - Interrupt status (intrstat 1 10) - Exit code 127 [2016-02-11 14:32:59,489] - [remote_util] [140167490725696] - INFO - Processor status (mpstat 1 10) - Exit code 127 [2016-02-11 14:32:59,489] - [remote_util] [140167490725696] - INFO - System log (cat /var/adm/messages) - Exit code 1 [2016-02-11 14:32:59,490] - [remote_util] [140167490725696] - INFO - Raw /proc/uptime (cat /proc/uptime) - OK [2016-02-11 14:32:59,490] - [remote_util] [140167490725696] - INFO - All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 [2016-02-11 14:32:59,490] - [remote_util] [140167490725696] - INFO - Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK [2016-02-11 14:32:59,490] - [remote_util] [140167490725696] - INFO - Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK [2016-02-11 14:32:59,490] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --hardware) - Exit code 127 [2016-02-11 14:32:59,490] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --show) - Exit code 127 [2016-02-11 14:32:59,491] - [remote_util] [140167490725696] - INFO - NUMA data (cat /sys/devices/system/node/node*/numastat) - OK [2016-02-11 14:32:59,491] - [remote_util] [140167490725696] - INFO - Kernel log buffer (dmesg -H || dmesg) - OK [2016-02-11 14:32:59,491] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:32:59,491] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:32:59,491] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - Network statistics (netstat -s) - OK [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - Full raw netstat (cat /proc/net/netstat) - OK [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - Found dump-guts: /opt/couchbase/bin/dump-guts [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... [2016-02-11 14:32:59,492] - [remote_util] [140167490725696] - INFO - product diag header () - OK [2016-02-11 14:32:59,493] - [remote_util] [140167490725696] - INFO - Found escript-wrapper: /opt/couchbase/bin/escript-wrapper [2016-02-11 14:32:59,493] - [remote_util] [140167490725696] - INFO - Found dump-stats: /opt/couchbase/bin/dump-stats [2016-02-11 14:32:59,493] - [remote_util] [140167490725696] - INFO - Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK [2016-02-11 14:32:59,493] - [remote_util] [140167490725696] - INFO - Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:32:59,493] - [remote_util] [140167490725696] - INFO - Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:32:59,493] - [remote_util] [140167490725696] - INFO - couch_dbinfo (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-name', '*.couch.*', '-exec', 'couch_dbinfo', '{}', '+']) - OK [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Version file (cat '/opt/couchbase/VERSION.txt') - OK [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.txt') - Exit code 1 [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.xml') - OK [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Couchbase config () - OK [2016-02-11 14:32:59,494] - [remote_util] [140167490725696] - INFO - Couchbase static config () - OK [2016-02-11 14:32:59,495] - [remote_util] [140167490725696] - INFO - Raw ns_log () - OK [2016-02-11 14:32:59,495] - [remote_util] [140167490725696] - INFO - Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK [2016-02-11 14:32:59,495] - [remote_util] [140167490725696] - INFO - Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK [2016-02-11 14:32:59,495] - [remote_util] [140167490725696] - INFO - couchbase diags (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag?noLogs=1']) - OK [2016-02-11 14:32:59,495] - [remote_util] [140167490725696] - INFO - master events (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/masterEvents?o=1']) - OK [2016-02-11 14:32:59,496] - [remote_util] [140167490725696] - INFO - ale configuration (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/ale']) - OK [2016-02-11 14:32:59,496] - [remote_util] [140167490725696] - INFO - couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK [2016-02-11 14:32:59,496] - [remote_util] [140167490725696] - INFO - couchbase logs (info.log) (cbbrowse_logs info.log) - OK [2016-02-11 14:32:59,496] - [remote_util] [140167490725696] - INFO - couchbase logs (error.log) (cbbrowse_logs error.log) - OK [2016-02-11 14:32:59,497] - [remote_util] [140167490725696] - INFO - couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK [2016-02-11 14:32:59,497] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK [2016-02-11 14:32:59,497] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK [2016-02-11 14:32:59,497] - [remote_util] [140167490725696] - INFO - couchbase logs (views.log) (cbbrowse_logs views.log) - OK [2016-02-11 14:32:59,497] - [remote_util] [140167490725696] - INFO - couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK [2016-02-11 14:32:59,498] - [remote_util] [140167490725696] - INFO - couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK [2016-02-11 14:32:59,498] - [remote_util] [140167490725696] - INFO - couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK [2016-02-11 14:32:59,498] - [remote_util] [140167490725696] - INFO - couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK [2016-02-11 14:32:59,498] - [remote_util] [140167490725696] - INFO - couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 [2016-02-11 14:32:59,499] - [remote_util] [140167490725696] - INFO - couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK [2016-02-11 14:32:59,500] - [remote_util] [140167490725696] - INFO - couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 [2016-02-11 14:32:59,500] - [remote_util] [140167490725696] - INFO - couchbase logs (fts.log) (cbbrowse_logs fts.log) - Exit code 1 [2016-02-11 14:32:59,500] - [remote_util] [140167490725696] - INFO - couchbase logs (metakv.log) (cbbrowse_logs metakv.log) - OK [2016-02-11 14:32:59,500] - [remote_util] [140167490725696] - INFO - memcached stats all (['cbstats', '-a', '127.0.0.1:11209', 'all', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,501] - [remote_util] [140167490725696] - INFO - memcached stats allocator (['cbstats', '-a', '127.0.0.1:11209', 'allocator', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,501] - [remote_util] [140167490725696] - INFO - memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11209', 'checkpoint', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,501] - [remote_util] [140167490725696] - INFO - memcached stats config (['cbstats', '-a', '127.0.0.1:11209', 'config', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,501] - [remote_util] [140167490725696] - INFO - memcached stats dcp (['cbstats', '-a', '127.0.0.1:11209', 'dcp', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,502] - [remote_util] [140167490725696] - INFO - memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11209', 'dcpagg', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,502] - [remote_util] [140167490725696] - INFO - memcached stats ['diskinfo', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'diskinfo', 'detail', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,502] - [remote_util] [140167490725696] - INFO - memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11209', 'dispatcher', 'logs', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,502] - [remote_util] [140167490725696] - INFO - memcached stats failovers (['cbstats', '-a', '127.0.0.1:11209', 'failovers', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,503] - [remote_util] [140167490725696] - INFO - memcached stats ['hash', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'hash', 'detail', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,503] - [remote_util] [140167490725696] - INFO - memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11209', 'kvstore', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,503] - [remote_util] [140167490725696] - INFO - memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11209', 'kvtimings', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,503] - [remote_util] [140167490725696] - INFO - memcached stats memory (['cbstats', '-a', '127.0.0.1:11209', 'memory', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,504] - [remote_util] [140167490725696] - INFO - memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11209', 'prev-vbucket', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,504] - [remote_util] [140167490725696] - INFO - memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11209', 'runtimes', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,504] - [remote_util] [140167490725696] - INFO - memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11209', 'scheduler', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,504] - [remote_util] [140167490725696] - INFO - memcached stats tap (['cbstats', '-a', '127.0.0.1:11209', 'tap', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,504] - [remote_util] [140167490725696] - INFO - memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11209', 'tapagg', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,504] - [remote_util] [140167490725696] - INFO - memcached stats timings (['cbstats', '-a', '127.0.0.1:11209', 'timings', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,505] - [remote_util] [140167490725696] - INFO - memcached stats uuid (['cbstats', '-a', '127.0.0.1:11209', 'uuid', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,505] - [remote_util] [140167490725696] - INFO - memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11209', 'vbucket', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,505] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-details', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,505] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-seqno (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-seqno', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,505] - [remote_util] [140167490725696] - INFO - memcached stats warmup (['cbstats', '-a', '127.0.0.1:11209', 'warmup', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,506] - [remote_util] [140167490725696] - INFO - memcached stats workload (['cbstats', '-a', '127.0.0.1:11209', 'workload', '-b', '_admin', '-p', 'b7f3e080ab51e5d7250dc1e459f55bab']) - Exit code 1 [2016-02-11 14:32:59,506] - [remote_util] [140167490725696] - INFO - memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11209', '-u', '_admin', '-P', 'b7f3e080ab51e5d7250dc1e459f55bab', 'connections']) - OK [2016-02-11 14:32:59,506] - [remote_util] [140167490725696] - INFO - replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK [2016-02-11 14:32:59,506] - [remote_util] [140167490725696] - INFO - moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 [2016-02-11 14:32:59,506] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', 'b7f3e080ab51e5d7250dc1e459f55bab', '-h', '127.0.0.1:11209', '-v']) - OK [2016-02-11 14:32:59,506] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', 'b7f3e080ab51e5d7250dc1e459f55bab', '-h', '127.0.0.1:11209', '-v', 'subdoc_execute']) - OK [2016-02-11 14:32:59,507] - [remote_util] [140167490725696] - INFO - stats archives (['escript', '/opt/couchbase/bin/escript-wrapper', '--initargs-path', '/opt/couchbase/var/lib/couchbase/initargs', '--', '/opt/couchbase/bin/dump-stats', '/opt/couchbase/var/lib/couchbase/stats']) - OK [2016-02-11 14:32:59,507] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/couchbase.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/couchbase.log [2016-02-11 14:32:59,507] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ddocs.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ddocs.log [2016-02-11 14:32:59,507] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.error.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.error.log [2016-02-11 14:32:59,508] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.ns_couchdb.log [2016-02-11 14:32:59,508] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/stats.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/stats.log [2016-02-11 14:32:59,508] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/memcached.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/memcached.log [2016-02-11 14:32:59,508] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.ssl_proxy.log [2016-02-11 14:32:59,509] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.indexer.log [2016-02-11 14:32:59,509] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.xdcr_errors.log [2016-02-11 14:32:59,509] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.fts.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.fts.log [2016-02-11 14:32:59,509] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.http_access_internal.log [2016-02-11 14:32:59,509] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/syslog.tar.gz -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/syslog.tar.gz [2016-02-11 14:32:59,509] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.reports.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.reports.log [2016-02-11 14:32:59,510] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.query.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.query.log [2016-02-11 14:32:59,510] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.couchdb.log [2016-02-11 14:32:59,510] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.goxdcr.log [2016-02-11 14:32:59,510] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/master_events.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/master_events.log [2016-02-11 14:32:59,510] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.info.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.info.log [2016-02-11 14:32:59,511] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/diag.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/diag.log [2016-02-11 14:32:59,511] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.stats.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.stats.log [2016-02-11 14:32:59,511] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.xdcr.log [2016-02-11 14:32:59,511] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.views.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.views.log [2016-02-11 14:32:59,511] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ini.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ini.log [2016-02-11 14:32:59,511] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/stats_archives.json -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/stats_archives.json [2016-02-11 14:32:59,512] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.mapreduce_errors.log [2016-02-11 14:32:59,512] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.projector.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.projector.log [2016-02-11 14:32:59,512] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.metakv.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.metakv.log [2016-02-11 14:32:59,512] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.xdcr_trace.log [2016-02-11 14:32:59,512] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.debug.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.debug.log [2016-02-11 14:32:59,512] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.babysitter.log [2016-02-11 14:32:59,513] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmpqLvwND/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.107.194_20160211-223157/ns_server.http_access.log [2016-02-11 14:32:59,823] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.194: rm -f /root/172.23.107.194-20160211-1431-diag.zip [2016-02-11 14:32:59,845] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:32:59,848] - [remote_util] [140167490725696] - INFO - connecting to 172.23.107.38 with username : root password : couchbase ssh_key: [2016-02-11 14:33:00,022] - [remote_util] [140167490725696] - INFO - Connected to 172.23.107.38 [2016-02-11 14:33:00,525] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.38: /opt/couchbase/bin/cbcollect_info 172.23.107.38-20160211-1432-diag.zip [2016-02-11 14:34:03,721] - [remote_util] [140167490725696] - INFO - command executed successfully [2016-02-11 14:34:03,721] - [remote_util] [140167490725696] - INFO - uname (uname -a) - OK [2016-02-11 14:34:03,722] - [remote_util] [140167490725696] - INFO - time and TZ (date; date -u) - OK [2016-02-11 14:34:03,722] - [remote_util] [140167490725696] - INFO - ntp time (ntpdate -q pool.ntp.org || nc time.nist.gov 13 || netcat time.nist.gov 13) - OK [2016-02-11 14:34:03,722] - [remote_util] [140167490725696] - INFO - ntp peers (ntpq -p) - OK [2016-02-11 14:34:03,722] - [remote_util] [140167490725696] - INFO - raw /etc/sysconfig/clock (cat /etc/sysconfig/clock) - OK [2016-02-11 14:34:03,722] - [remote_util] [140167490725696] - INFO - raw /etc/timezone (cat /etc/timezone) - Exit code 1 [2016-02-11 14:34:03,722] - [remote_util] [140167490725696] - INFO - System Hardware (lshw -json || lshw) - Exit code 127 [2016-02-11 14:34:03,723] - [remote_util] [140167490725696] - INFO - Process list snapshot (export TERM=''; top -Hb -n1 || top -H n1) - OK [2016-02-11 14:34:03,723] - [remote_util] [140167490725696] - INFO - Process list (ps -AwwL -o user,pid,lwp,ppid,nlwp,pcpu,maj_flt,min_flt,pri,nice,vsize,rss,tty,stat,wchan:12,start,bsdtime,command) - OK [2016-02-11 14:34:03,723] - [remote_util] [140167490725696] - INFO - Raw /proc/vmstat (cat /proc/vmstat) - OK [2016-02-11 14:34:03,723] - [remote_util] [140167490725696] - INFO - Raw /proc/mounts (cat /proc/mounts) - OK [2016-02-11 14:34:03,723] - [remote_util] [140167490725696] - INFO - Raw /proc/partitions (cat /proc/partitions) - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Raw /proc/diskstats (cat /proc/diskstats; echo '') - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 1.000000 seconds - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Taking sample 3 after 1.000000 seconds - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Taking sample 4 after 1.000000 seconds - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Taking sample 5 after 1.000000 seconds - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Taking sample 6 after 1.000000 seconds - OK [2016-02-11 14:34:03,724] - [remote_util] [140167490725696] - INFO - Taking sample 7 after 1.000000 seconds - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Taking sample 8 after 1.000000 seconds - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Taking sample 9 after 1.000000 seconds - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Taking sample 10 after 1.000000 seconds - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Raw /proc/interrupts (cat /proc/interrupts) - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Swap configuration (free -t) - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Swap configuration (swapon -s) - OK [2016-02-11 14:34:03,725] - [remote_util] [140167490725696] - INFO - Kernel modules (lsmod) - OK [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/redhat-release) - OK [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Distro version (lsb_release -a) - OK [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/SuSE-release) - Exit code 1 [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Distro version (cat /etc/issue) - OK [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Installed software (rpm -qa) - OK [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Installed software (COLUMNS=300 dpkg -l) - Exit code 127 [2016-02-11 14:34:03,726] - [remote_util] [140167490725696] - INFO - Extended iostat (iostat -x -p ALL 1 10 || iostat -x 1 10) - Exit code 127 [2016-02-11 14:34:03,727] - [remote_util] [140167490725696] - INFO - Core dump settings (find /proc/sys/kernel -type f -name '*core*' -print -exec cat '{}' ';') - OK [2016-02-11 14:34:03,727] - [remote_util] [140167490725696] - INFO - sysctl settings (sysctl -a) - OK [2016-02-11 14:34:03,727] - [remote_util] [140167490725696] - INFO - Relevant lsof output (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -r -- lsof -n -p) - Exit code 127 [2016-02-11 14:34:03,727] - [remote_util] [140167490725696] - INFO - LVM info (lvdisplay) - OK [2016-02-11 14:34:03,727] - [remote_util] [140167490725696] - INFO - LVM info (vgdisplay) - OK [2016-02-11 14:34:03,727] - [remote_util] [140167490725696] - INFO - LVM info (pvdisplay) - OK [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Network configuration (ifconfig -a) - OK [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Taking sample 2 after 10.000000 seconds - OK [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Network configuration (echo link addr neigh rule route netns | xargs -n1 -- sh -x -c 'ip $1 list' --) - Exit code 124 [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Raw /proc/net/dev (cat /proc/net/dev) - OK [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Network link statistics (ip -s link) - OK [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Network status (netstat -anp || netstat -an) - OK [2016-02-11 14:34:03,728] - [remote_util] [140167490725696] - INFO - Network routing table (netstat -rn) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Network socket statistics (ss -an) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Extended socket statistics (ss -an --info --processes) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Arp cache (arp -na) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Iptables dump (iptables-save) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Raw /etc/hosts (cat /etc/hosts) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Raw /etc/resolv.conf (cat /etc/resolv.conf) - OK [2016-02-11 14:34:03,729] - [remote_util] [140167490725696] - INFO - Raw /etc/nsswitch.conf (cat /etc/nsswitch.conf) - OK [2016-02-11 14:34:03,730] - [remote_util] [140167490725696] - INFO - Filesystem (df -ha) - OK [2016-02-11 14:34:03,730] - [remote_util] [140167490725696] - INFO - System activity reporter (sar 1 10) - Exit code 127 [2016-02-11 14:34:03,730] - [remote_util] [140167490725696] - INFO - System paging activity (vmstat 1 10) - OK [2016-02-11 14:34:03,730] - [remote_util] [140167490725696] - INFO - System uptime (uptime) - OK [2016-02-11 14:34:03,731] - [remote_util] [140167490725696] - INFO - couchbase user definition (getent passwd couchbase) - OK [2016-02-11 14:34:03,731] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:34:03,731] - [remote_util] [140167490725696] - INFO - couchbase user limits (su couchbase -c "ulimit -a") - Exit code 1 [2016-02-11 14:34:03,731] - [remote_util] [140167490725696] - INFO - Interrupt status (intrstat 1 10) - Exit code 127 [2016-02-11 14:34:03,731] - [remote_util] [140167490725696] - INFO - Processor status (mpstat 1 10) - Exit code 127 [2016-02-11 14:34:03,731] - [remote_util] [140167490725696] - INFO - System log (cat /var/adm/messages) - Exit code 1 [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - Raw /proc/uptime (cat /proc/uptime) - OK [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - All logs (tar cz /var/log/syslog* /var/log/dmesg /var/log/messages* /var/log/daemon* /var/log/debug* /var/log/kern.log* 2>/dev/null) - Exit code 2 [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - Relevant proc data (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; cat /proc/$1/status; cat /proc/$1/limits; cat /proc/$1/smaps; cat /proc/$1/numa_maps; cat /proc/$1/task/*/sched; echo' --) - OK [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - Processes' environment (echo moxi memcached beam.smp couch_compact godu sigar_port cbq-engine indexer projector goxdcr fts | xargs -n1 pgrep | xargs -n1 -- sh -c 'echo $1; ( cat /proc/$1/environ | tr \\0 \\n ); echo' --) - OK [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --hardware) - Exit code 127 [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - NUMA data (numactl --show) - Exit code 127 [2016-02-11 14:34:03,732] - [remote_util] [140167490725696] - INFO - NUMA data (cat /sys/devices/system/node/node*/numastat) - OK [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Kernel log buffer (dmesg -H || dmesg) - OK [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/enabled) - Exit code 1 [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Transparent Huge Pages data (cat /sys/kernel/mm/redhat_transparent_hugepage/defrag) - Exit code 1 [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Network statistics (netstat -s) - OK [2016-02-11 14:34:03,733] - [remote_util] [140167490725696] - INFO - Full raw netstat (cat /proc/net/netstat) - OK [2016-02-11 14:34:03,734] - [remote_util] [140167490725696] - INFO - CPU throttling info (echo /sys/devices/system/cpu/cpu*/thermal_throttle/* | xargs -n1 -- sh -c 'echo $1; cat $1' --) - Exit code 123 [2016-02-11 14:34:03,734] - [remote_util] [140167490725696] - INFO - Found dump-guts: /opt/couchbase/bin/dump-guts [2016-02-11 14:34:03,734] - [remote_util] [140167490725696] - INFO - Checking for server guts in /opt/couchbase/var/lib/couchbase/initargs... [2016-02-11 14:34:03,734] - [remote_util] [140167490725696] - INFO - product diag header () - OK [2016-02-11 14:34:03,734] - [remote_util] [140167490725696] - INFO - Found escript-wrapper: /opt/couchbase/bin/escript-wrapper [2016-02-11 14:34:03,734] - [remote_util] [140167490725696] - INFO - Found dump-stats: /opt/couchbase/bin/dump-stats [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - Directory structure (['ls', '-lRai', '/opt/couchbase']) - OK [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - Database directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - Index directory structure (['ls', '-lRai', '/opt/couchbase/var/lib/couchbase/data']) - OK [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - couch_dbinfo (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-name', '*.couch.*', '-exec', 'couch_dbinfo', '{}', '+']) - OK [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - Database directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - Index directory filefrag info (['find', '/opt/couchbase/var/lib/couchbase/data', '-type', 'f', '-exec', 'filefrag', '-v', '{}', '+']) - OK [2016-02-11 14:34:03,735] - [remote_util] [140167490725696] - INFO - Version file (cat '/opt/couchbase/VERSION.txt') - OK [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.txt') - Exit code 1 [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Manifest file (cat '/opt/couchbase/manifest.xml') - OK [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Couchbase config () - OK [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Couchbase static config () - OK [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Raw ns_log () - OK [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Memcached logs (['sh', '-c', 'cd "$1"; for file in $(ls -tr memcached.log.*); do cat "$file"; done', '--', '/opt/couchbase/var/lib/couchbase/logs']) - OK [2016-02-11 14:34:03,736] - [remote_util] [140167490725696] - INFO - Ini files (['sh', '-c', 'for i in "$@"; do echo "file: $i"; cat "$i"; done', '--', '/opt/couchbase/etc/couchdb/default.ini', '/opt/couchbase/etc/couchdb/default.d/capi.ini', '/opt/couchbase/etc/couchdb/default.d/geocouch.ini', '/opt/couchbase/etc/couchdb/local.ini']) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - couchbase diags (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag?noLogs=1']) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - master events (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/masterEvents?o=1']) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - ale configuration (['curl', '-sS', '--proxy', '', '-u', '@:*****', 'http://127.0.0.1:8091/diag/ale']) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - couchbase logs (debug.log) (cbbrowse_logs debug.log) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - couchbase logs (info.log) (cbbrowse_logs info.log) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - couchbase logs (error.log) (cbbrowse_logs error.log) - OK [2016-02-11 14:34:03,737] - [remote_util] [140167490725696] - INFO - couchbase logs (couchdb.log) (cbbrowse_logs couchdb.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr.log) (cbbrowse_logs xdcr.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_errors.log) (cbbrowse_logs xdcr_errors.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (views.log) (cbbrowse_logs views.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (mapreduce_errors.log) (cbbrowse_logs mapreduce_errors.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (stats.log) (cbbrowse_logs stats.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (babysitter.log) (cbbrowse_logs babysitter.log) - OK [2016-02-11 14:34:03,738] - [remote_util] [140167490725696] - INFO - couchbase logs (ssl_proxy.log) (cbbrowse_logs ssl_proxy.log) - OK [2016-02-11 14:34:03,739] - [remote_util] [140167490725696] - INFO - couchbase logs (reports.log) (cbbrowse_logs reports.log) - OK [2016-02-11 14:34:03,739] - [remote_util] [140167490725696] - INFO - couchbase logs (xdcr_trace.log) (cbbrowse_logs xdcr_trace.log) - OK [2016-02-11 14:34:03,739] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access.log) (cbbrowse_logs http_access.log) - OK [2016-02-11 14:34:03,739] - [remote_util] [140167490725696] - INFO - couchbase logs (http_access_internal.log) (cbbrowse_logs http_access_internal.log) - OK [2016-02-11 14:34:03,739] - [remote_util] [140167490725696] - INFO - couchbase logs (ns_couchdb.log) (cbbrowse_logs ns_couchdb.log) - OK [2016-02-11 14:34:03,739] - [remote_util] [140167490725696] - INFO - couchbase logs (goxdcr.log) (cbbrowse_logs goxdcr.log) - OK [2016-02-11 14:34:03,740] - [remote_util] [140167490725696] - INFO - couchbase logs (query.log) (cbbrowse_logs query.log) - Exit code 1 [2016-02-11 14:34:03,740] - [remote_util] [140167490725696] - INFO - couchbase logs (projector.log) (cbbrowse_logs projector.log) - OK [2016-02-11 14:34:03,740] - [remote_util] [140167490725696] - INFO - couchbase logs (indexer.log) (cbbrowse_logs indexer.log) - Exit code 1 [2016-02-11 14:34:03,740] - [remote_util] [140167490725696] - INFO - couchbase logs (fts.log) (cbbrowse_logs fts.log) - Exit code 1 [2016-02-11 14:34:03,740] - [remote_util] [140167490725696] - INFO - couchbase logs (metakv.log) (cbbrowse_logs metakv.log) - OK [2016-02-11 14:34:03,740] - [remote_util] [140167490725696] - INFO - memcached stats all (['cbstats', '-a', '127.0.0.1:11209', 'all', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,741] - [remote_util] [140167490725696] - INFO - memcached stats allocator (['cbstats', '-a', '127.0.0.1:11209', 'allocator', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,741] - [remote_util] [140167490725696] - INFO - memcached stats checkpoint (['cbstats', '-a', '127.0.0.1:11209', 'checkpoint', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,741] - [remote_util] [140167490725696] - INFO - memcached stats config (['cbstats', '-a', '127.0.0.1:11209', 'config', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,741] - [remote_util] [140167490725696] - INFO - memcached stats dcp (['cbstats', '-a', '127.0.0.1:11209', 'dcp', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,741] - [remote_util] [140167490725696] - INFO - memcached stats dcpagg (['cbstats', '-a', '127.0.0.1:11209', 'dcpagg', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,741] - [remote_util] [140167490725696] - INFO - memcached stats ['diskinfo', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'diskinfo', 'detail', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,742] - [remote_util] [140167490725696] - INFO - memcached stats ['dispatcher', 'logs'] (['cbstats', '-a', '127.0.0.1:11209', 'dispatcher', 'logs', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,742] - [remote_util] [140167490725696] - INFO - memcached stats failovers (['cbstats', '-a', '127.0.0.1:11209', 'failovers', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,742] - [remote_util] [140167490725696] - INFO - memcached stats ['hash', 'detail'] (['cbstats', '-a', '127.0.0.1:11209', 'hash', 'detail', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,742] - [remote_util] [140167490725696] - INFO - memcached stats kvstore (['cbstats', '-a', '127.0.0.1:11209', 'kvstore', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,742] - [remote_util] [140167490725696] - INFO - memcached stats kvtimings (['cbstats', '-a', '127.0.0.1:11209', 'kvtimings', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,742] - [remote_util] [140167490725696] - INFO - memcached stats memory (['cbstats', '-a', '127.0.0.1:11209', 'memory', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,743] - [remote_util] [140167490725696] - INFO - memcached stats prev-vbucket (['cbstats', '-a', '127.0.0.1:11209', 'prev-vbucket', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,743] - [remote_util] [140167490725696] - INFO - memcached stats runtimes (['cbstats', '-a', '127.0.0.1:11209', 'runtimes', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,743] - [remote_util] [140167490725696] - INFO - memcached stats scheduler (['cbstats', '-a', '127.0.0.1:11209', 'scheduler', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,743] - [remote_util] [140167490725696] - INFO - memcached stats tap (['cbstats', '-a', '127.0.0.1:11209', 'tap', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,743] - [remote_util] [140167490725696] - INFO - memcached stats tapagg (['cbstats', '-a', '127.0.0.1:11209', 'tapagg', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats timings (['cbstats', '-a', '127.0.0.1:11209', 'timings', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats uuid (['cbstats', '-a', '127.0.0.1:11209', 'uuid', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats vbucket (['cbstats', '-a', '127.0.0.1:11209', 'vbucket', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-details (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-details', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats vbucket-seqno (['cbstats', '-a', '127.0.0.1:11209', 'vbucket-seqno', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats warmup (['cbstats', '-a', '127.0.0.1:11209', 'warmup', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,744] - [remote_util] [140167490725696] - INFO - memcached stats workload (['cbstats', '-a', '127.0.0.1:11209', 'workload', '-b', '_admin', '-p', '552caed11caac29a1024be31d1457eba']) - Exit code 1 [2016-02-11 14:34:03,745] - [remote_util] [140167490725696] - INFO - memcached mcstat connections (['mcstat', '-h', '127.0.0.1:11209', '-u', '_admin', '-P', '552caed11caac29a1024be31d1457eba', 'connections']) - OK [2016-02-11 14:34:03,745] - [remote_util] [140167490725696] - INFO - replication docs (/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1) (['couch_dbdump', '/opt/couchbase/var/lib/couchbase/data/_replicator.couch.1']) - OK [2016-02-11 14:34:03,745] - [remote_util] [140167490725696] - INFO - moxi stats (port 11211) (echo stats proxy | nc 127.0.0.1 11211) - Exit code 127 [2016-02-11 14:34:03,745] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', '552caed11caac29a1024be31d1457eba', '-h', '127.0.0.1:11209', '-v']) - OK [2016-02-11 14:34:03,745] - [remote_util] [140167490725696] - INFO - mctimings (['mctimings', '-u', '_admin', '-P', '552caed11caac29a1024be31d1457eba', '-h', '127.0.0.1:11209', '-v', 'subdoc_execute']) - OK [2016-02-11 14:34:03,745] - [remote_util] [140167490725696] - INFO - stats archives (['escript', '/opt/couchbase/bin/escript-wrapper', '--initargs-path', '/opt/couchbase/var/lib/couchbase/initargs', '--', '/opt/couchbase/bin/dump-stats', '/opt/couchbase/var/lib/couchbase/stats']) - OK [2016-02-11 14:34:03,746] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/couchbase.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/couchbase.log [2016-02-11 14:34:03,746] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ddocs.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ddocs.log [2016-02-11 14:34:03,746] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.error.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.error.log [2016-02-11 14:34:03,746] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.ns_couchdb.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.ns_couchdb.log [2016-02-11 14:34:03,746] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/stats.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/stats.log [2016-02-11 14:34:03,746] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/memcached.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/memcached.log [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.ssl_proxy.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.ssl_proxy.log [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.indexer.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.indexer.log [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.xdcr_errors.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.xdcr_errors.log [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.fts.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.fts.log [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.http_access_internal.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.http_access_internal.log [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/syslog.tar.gz -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/syslog.tar.gz [2016-02-11 14:34:03,747] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.reports.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.reports.log [2016-02-11 14:34:03,748] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.query.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.query.log [2016-02-11 14:34:03,748] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.couchdb.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.couchdb.log [2016-02-11 14:34:03,748] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.goxdcr.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.goxdcr.log [2016-02-11 14:34:03,748] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/master_events.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/master_events.log [2016-02-11 14:34:03,748] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.info.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.info.log [2016-02-11 14:34:03,748] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/diag.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/diag.log [2016-02-11 14:34:03,749] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.stats.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.stats.log [2016-02-11 14:34:03,749] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.xdcr.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.xdcr.log [2016-02-11 14:34:03,749] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.views.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.views.log [2016-02-11 14:34:03,749] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ini.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ini.log [2016-02-11 14:34:03,749] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/stats_archives.json -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/stats_archives.json [2016-02-11 14:34:03,749] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.mapreduce_errors.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.mapreduce_errors.log [2016-02-11 14:34:03,750] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.projector.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.projector.log [2016-02-11 14:34:03,750] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.metakv.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.metakv.log [2016-02-11 14:34:03,750] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.xdcr_trace.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.xdcr_trace.log [2016-02-11 14:34:03,750] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.debug.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.debug.log [2016-02-11 14:34:03,750] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.babysitter.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.babysitter.log [2016-02-11 14:34:03,750] - [remote_util] [140167490725696] - INFO - adding: /tmp/tmp03q1Fs/ns_server.http_access.log -> cbcollect_info_ns_1@172.23.107.38_20160211-223300/ns_server.http_access.log [2016-02-11 14:34:03,940] - [remote_util] [140167490725696] - INFO - running command.raw on 172.23.107.38: rm -f /root/172.23.107.38-20160211-1432-diag.zip [2016-02-11 14:34:03,960] - [remote_util] [140167490725696] - INFO - command executed successfully