One node is slow during indexing

I don't know whether this is an environmental problem or not. On Pavels performance run with 4 nodes, one node is slow, one half way slow and two normal. You can find the logs of the slow run here [1].

If you look at the current ShowFast graph [2] of the "Initial index (min), 1 bucket x 50M x 2KB, DGM, 4 x 1 views, no mutations" run ("Linux", "View Indexing" -> "Initial", second graph), it's way slower in the build 956 than in the 928 (46.1s vs. 22.6s). When looking at the logs, it's node *.31 that's way slower. It is either ep-engine not providing the UPR stream messages fast enough, or the view-engine consuming them slowly.

This node has been shown to be slow in several tests, so it might even be a problem in the environment (like a slow disk).

Here's the analysis from the 4 nodes, where you can see that one is clearly way slower. The numbers on the right are the seconds between the "Backfill complete" and "Stream closing" message, the left number is how often it occurred:

cat cbcollect_info_ns_1@172.23.100.31_20140714-125849/memcached.log|grep 'Backfill complete|Stream closing'|grep '_design/A'|cut -d ' ' -f 4|xargs -I {} date --date={} +'%s'|awk '

{p=$1; getline; print $1-p}' > /tmp/31
vmx@emil$ cat cbcollect_info_ns_1@172.23.100.29_20140714-125849/memcached.log|grep 'Backfill complete|Stream closing'|grep '_design/A'|cut -d ' ' -f 4|xargs -I {} date --date={} +'%s'|awk '{p=$1; getline; print $1-p}

'|sort -n|uniq -c
301 2
208 3
1 4
1 5
1 8
vmx@emil$ cat cbcollect_info_ns_1@172.23.100.30_20140714-125849/memcached.log|grep 'Backfill complete|Stream closing'|grep '_design/A'|cut -d ' ' -f 4|xargs -I {} date --date={} +'%s'|awk '

{p=$1; getline; print $1-p}'|sort -n|uniq -c
169 2
87 3
16 4
82 5
119 6
28 7
9 8
2 9
vmx@emil$ cat cbcollect_info_ns_1@172.23.100.31_20140714-125849/memcached.log|grep 'Backfill complete|Stream closing'|grep '_design/A'|cut -d ' ' -f 4|xargs -I {} date --date={} +'%s'|awk '{p=$1; getline; print $1-p}

'|sort -n|uniq -c
9 5
41 6
146 7
124 8
76 9
67 10
29 11
15 12
3 13
1 14
1 16
vmx@emil$ cat cbcollect_info_ns_1@172.23.100.32_20140714-125849/memcached.log|grep 'Backfill complete|Stream closing'|grep '_design/A'|cut -d ' ' -f 4|xargs -I {} date --date={} +'%s'|awk '

{p=$1; getline; print $1-p}

'|sort -n|uniq -c
317 2
195 3

[1] http://localhost:3000/job/leto/298/
[2] http://showfast.sc.couchbase.com/#/timeline