Details
-
Bug
-
Resolution: Incomplete
-
Major
-
2.0-beta
-
Security Level: Public
-
build-1848
Description
Running about 300 queries/sec against 2 buckets
- Started rebalance
- Rebalance fails with error in vbucket_mover
...
<0.7975.178> exited with {unexpected_exit,
{'EXIT',<0.8005.178>,
{{badmatch,
[{'EXIT',
{timeout, {gen_server,call, [<15200.2771.3094>,had_backfill,30000]}}}]},
{ns_single_vbucket_mover, '-wait_backfill_determination/1-fun-1-',1}
[]}}}
- during rebalance I see a lot of these errors:
Stacktrace: [
{couch_view_merger,simple_set_view_query,3},
,
,
]
[couchdb:error,2012-10-16T8:43:02.960,ns_1@10.6.2.66:<0.4247.177>:couch_log:error:42]Uncaught error in HTTP request: {throw,{error,set_view_outdated}}
- Manual queries throws inconsistent state and looks like it's pointing to missing partition [6]:
http://10.6.2.68:8092/default/_design/d1/_view/v1?connection_timeout=60000&limit=10&skip=0
[couchdb:info,2012-10-16T15:59:14.035,ns_1@10.6.2.66:<0.3334.193>:couch_log:info:39]Set view `default`, group `_design/d1`, missing partitions: [6]
[couchdb:error,2012-10-16T15:59:19.043,ns_1@10.6.2.66:<0.7960.182>:couch_log:error:42]Uncaught error in HTTP request: {throw,{error,inconsistent_state}}
- cbstats shows that partition [6] is neither active nor replica on10.6.2.66,
[root@pine-11802 logs]# /opt/couchbase/bin/cbstats localhost:11210 vbucket -b saslbucket -p password | grep vb_6:
[root@pine-11802 logs]#
*however the file exists in its data directory (could be we failed to move?)
/opt/couchbase/bin/couch_dbinfo /data/default/6.couch.12
DB Info (/data/default/6.couch.12)
file format version: 10
update_seq: 20614
doc count: 13566
deleted doc count: 112
data size: 13.16 MB
B-tree size: 1.25 MB
total disk size: 18.18 MB
*indexer doesn't expect this db to be here either
curl -s -u Administrator:password 'http://10.6.2.66:8092/_set_view/default/_design/d1/_info' | json_xs
"active_partitions" : [
0,
1,
2,
3,
4,
5,
7,
8,
9,
(full logs to be attached)