Verified on build 4.1.1-5904. This bug was fixed.

Merged into sherlock branch. Not sure if the fix version should still be 4.1.1 or 4.1.2 (or maybe something else).

I was able to recreate the issue.
When there is more than one node running the index service, then the problem is seen for a bucket which has an index on some nodes but not all.

We want stats samples from all nodes aggregated/merged before returning them to the caller.
When a node does not have an index defined for a bucket, it uses defaults for global stats entries.

The order of these global stat entries on the nodes which do not have an index defined is different from the ones on nodes which has an index defined for the bucket. The former is not sorted where as the later is. Because of the difference in the order, aggregate_stat_kv_pairs() does not aggregate/merge the sample from different nodes properly.

I have a potential fix for this.

=========================== Node with index defined ===============================
(n_2@127.0.0.1)2> rp(hd(ets:lookup('stats_archiver-@index-default-minute', ets:last('stats_archiver-@index-default-minute')))).
{1449259793107,
{stat_entry,1449259793107,
[{<<"index/Index1_default/data_size">>,61111},
{<<"index/Index1_default/disk_overhead_estimate">>,69468},
{<<"index/Index1_default/disk_size">>,131072},
{<<"index/Index1_default/frag_percent">>,53},
{<<"index/Index1_default/items_count">>,0},
{<<"index/Index1_default/num_docs_indexed">>,0.0},
{<<"index/Index1_default/num_docs_pending">>,0},
{<<"index/Index1_default/num_docs_queued">>,0},
{<<"index/Index1_default/num_requests">>,0.0},
{<<"index/Index1_default/num_rows_returned">>,0.0},
{<<"index/Index1_default/scan_bytes_read">>,0.0},
{<<"index/Index1_default/total_scan_duration">>,0.0},
{<<"index/data_size">>,61111},
{<<"index/disk_overhead_estimate">>,69468},
{<<"index/disk_size">>,131072},
{<<"index/frag_percent">>,53},
{<<"index/items_count">>,0},
{<<"index/num_docs_indexed">>,0.0},
{<<"index/num_docs_pending">>,0},
{<<"index/num_docs_queued">>,0},
{<<"index/num_requests">>,0.0},
{<<"index/num_rows_returned">>,0.0},
{<<"index/scan_bytes_read">>,0.0},
{<<"index/total_scan_duration">>,0.0}]}}
ok
(n_2@127.0.0.1)3>

=========================== Global stat entries on Node without index defined ===============================

(n_0@10.0.0.29)1> hd(ets:lookup('stats_archiver-@index-default-minute', ets:last('stats_archiver-@index-default-minute'))).
{1449259293108,
{stat_entry,1449259293108,
[{<<"index/disk_size">>,0},
{<<"index/data_size">>,0},
{<<"index/num_docs_pending">>,0},
{<<"index/num_docs_queued">>,0},
{<<"index/items_count">>,0},
{<<"index/frag_percent">>,0},
{<<"index/num_requests">>,0},
{<<"index/num_rows_returned">>,0},
{<<"index/num_docs_indexed">>,0},
{<<"index/scan_bytes_read">>,0},
{<<"index/total_scan_duration">>,0},
{<<"index/disk_overhead_estimate">>,0}]}}
(n_0@10.0.0.29)2>

Do not see the problem on my setup. Screenshot attached. Will check the logs.

Logs available here: https://s3.amazonaws.com/cb-support/malarky/10153/