[system test] view query hang during rebalance

Cluster information:

• 8 centos 6.2 64bit server with 4 cores CPU
• Each server has 32 GB RAM and 400 GB SSD disk.
• 24.8 GB RAM for couchbase server at each node
• SSD disk format ext4 on /data
• Each server has its own SSD drive, no disk sharing with other server.
• Create cluster with 6 nodes installed couchbase server 2.0.0-1781
• Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
• Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
• Enable consistent view on cluster (by default)

10.6.2.37
10.6.2.38
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43

• Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes
• Queries all 4 views from 2 docs

10.6.2.39
10.6.2.40

• Data path /data
• View path /data

Manifest info from build 1781
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1781-rel.rpm.manifest.xml

<manifest><remote name="couchbase" fetch="git://github.com/couchbase/"/><remote name="membase" fetch="git://github.com/membase/"/><remote name="apache" fetch="git://github.com/apache/"/><remote name="erlang" fetch="git://github.com/erlang/"/><default remote="couchbase" revision="master"/><project name="tlm" path="tlm" revision="3811f5fc95c9e6000fa4a17c7fc1f3648e108fe8"><copyfile src="Makefile.top" dest="Makefile"/></project><project name="bucket_engine" path="bucket_engine" revision="70b3624abc697b7d18bf3d57f331b7674544e1e7"/><project name="ep-engine" path="ep-engine" revision="d36950038a2e83a2df2a7de77653ad70648e702b"/><project name="libconflate" path="libconflate" revision="3cf7107eaa5b52b34cc9f887cf0e2edb3465988e"/><project name="libmemcached" path="libmemcached" revision="ca739a890349ac36dc79447e37da7caa9ae819f5" remote="membase"/><project name="libvbucket" path="libvbucket" revision="00d3763593c116e8e5d97aa0b646c42885727398"/><project name="membase-cli" path="membase-cli" revision="d3ef6e8bf3700ad138ca2affb94287aa5ad1cd3d" remote="membase"/><project name="memcached" path="memcached" revision="858731183b08cd6b72fa6e68c1fb4208cb87570d" remote="membase"/><project name="moxi" path="moxi" revision="52a5fa887bfff0bf719c4ee5f29634dd8707500e"/><project name="ns_server" path="ns_server" revision="d13618bc32ab1b368a1369cc25d64bce8a34bc6a"/><project name="portsigar" path="portsigar" revision="1bc865e1622fb93a3fe0d1a4cdf18eb97ed9d600"/><project name="sigar" path="sigar" revision="63a3cd1b316d2d4aa6dd31ce8fc66101b983e0b0"/><project name="couchbase-examples" path="couchbase-examples" revision="a9e2e92b1f4d12bbfb7f5873306b003d0b447093"/><project name="couchbase-python-client" path="couchbase-python-client" revision="3bd552c47ba0273d79159a5c808d6de2cd955dcb"/><project name="couchdb" path="couchdb" revision="804ef001394269d3c02e3eb1ef05ac3a02f86c54"/><project name="couchdbx-app" path="couchdbx-app" revision="f57578ee2f4ecbf58ac39f238bd23cb72013645e"/><project name="couchstore" path="couchstore" revision="29579bd47f7c916c43116722b8f4962b4ea9fff0"/><project name="geocouch" path="geocouch" revision="7782df1a53104e9c8bb9ef941a9b499bbc7cd61e"/><project name="mccouch" path="mccouch" revision="88701cc326bc3dde4ed072bb8441be83adcfb2a5"/><project name="testrunner" path="testrunner" revision="dcab423506883b6732ac25b30921745e79d2e221"/><project name="otp" path="otp" revision="b6dc1a844eab061d0a7153d46e7e68296f15a504" remote="erlang"/><project name="icu4c" path="icu4c" revision="26359393672c378f41f2103a8699c4357c894be7" remote="couchbase"/><project name="snappy" path="snappy" revision="5681dde156e9d07adbeeab79666c9a9d7a10ec95" remote="couchbase"/><project name="v8" path="v8" revision="447decb75060a106131ab4de934bcc374648e7f2" remote="couchbase"/><project name="gperftools" path="gperftools" revision="8f60ba949fb8576c530ef4be148bff97106ddc59" remote="couchbase"/><project name="pysqlite" path="pysqlite" revision="0ff6e32ea05037fddef1eb41a648f2a2141009ea" remote="couchbase"/>

• • Add 2 nodes: 39 and 40 and rebalance. During rebalance, reboot node 42 and 43. Rebalance failed as expected.
• After node finished warmup, rebalance again. Rebalance failed with bug MB-6490 on node 44.
• Failover node 44 and rebalance. During rebalance, query view from cluster. Query did not give any results nor timeout (with stale set to ok or false, even with connection_timeout=999999999)

http://10.6.2.37:8092/default/_design/d1/_view/v1?stale=ok&connection_timeout=60000&limit=10&skip=0
http://10.6.2.37:8092/default/_design/d1/_view/v1?stale=false&connection_timeout=60000&limit=10&skip=0

Set timeout to 999999999, still no results
curl -X GET 'http://saslbucket:password@10.6.2.39:8092/saslbucket/_design/d11/_view/v1?limit=10&stale=false&connection_timeout=999999999'

Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1781-view-query-hang-20121001-175054.tgz