Details
-
Bug
-
Resolution: Incomplete
-
Major
-
2.0-beta
-
Security Level: Public
-
None
-
centos 6.2 64bit
Description
Cluster information:
- 11 centos 6.2 64bit server with 4 cores CPU
- Each server has 10 GB RAM and 150 GB disk.
- 8 GB RAM for couchbase server at each node (80% total system memmories)
- Disk format ext3 on both data and root
- Each server has its own drive, no disk sharing with other server.
- Load 9 million items to both buckets and querying them continuously
- Initial indexing, so cpu a little heavy load
- Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
- Add one more doc d2 with 2 views to default bucket
- Create cluster with 10 nodes installed couchbase server 2.0.0-1663
10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23 - Data path /data
- View path /data
- The last run, I do swap rebalance remove node 13 and add node 26.
- Then node 26 failed due to physical failure. I failover node 26 and rebalance.
- Rebalance failed with known issue
MB-6497at the end of rebalance saslbucket - Do rebalance again. Rebalance failed due to couchbase server on node 22 shutdown (run out of space)
- Look at diags of log 22, I see write error for view replica
[couchdb:error,2012-09-05T9:25:20.626,ns_1@10.3.121.22:<0.21869.418>:couch_log:error:42]Set view `saslbucket`, replica group `_design/d11`, writer error
error: {{write_loop_died,{badmatch,
{gen_server,call,[<0.24573.403>,flush,infinity]}}
stacktrace: [{gen_server,call,3},
{couch_set_view_updater,write_changes,4},
{couch_set_view_updater,flush_writes,1},
{couch_set_view_updater,do_writes,1},
{couch_set_view_updater,'-update/7-fun-1-',14}]
[error_logger:error,2012-09-05T9:25:20.628,ns_1@10.3.121.22:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: couch_file:spawn_writer/2
pid: <0.24576.403>
registered_name: []
exception error: no match of right hand side value {error,enospc}
in function couch_file:write_blocks/3
in call from couch_file:writer_collect_chunks/5
ancestors: [<0.24573.403>,<0.15943.233>,<0.15934.233>,<0.15933.233>]
messages: []
links: <0.24573.403>,#Port<0.12904517>
dictionary: []
trap_exit: true
status: running
heap_size: 514229
stack_size: 24
reductions: 15097434098
neighbours:
Link to collect info of node 22
https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_0/201209/info_node22-20120905.zip