Details
Major Description
Setup
-----------------
1. Load 4M items on source cluster, setup a unidirectioanl replication from source to destination cluster.
2. Reboot one node on the source cluster.
Output/Error
-----------------------
Replication status shows failed but replication is continuing successfully.
Items are replicated successfully on the destination cluster.
The UI logs show "Replication failed" error messages [ Not sure if this ties up with the failed messages displayed]
Attached is the screenshot from the source cluster.
[xdcr:error] [2012-07-25 9:20:45] [ns_1@10.3.121.31:<0.20108.1>:xdc_replicator:handle_info:282] Worker <0.20128.1> died with reason: {function_clause,
[
,
{xdc_replicator_worker, maybe_flush_docs,3},
{xdc_replicator_worker, '-local_process_batch/6-fun-1-',3},
{lists,foldl,3},
{xdc_replicator_worker, local_process_batch,6},
{xdc_replicator_worker, queue_fetch_loop,5}]}
[xdcr:error] [2012-07-25 9:20:45] [ns_1@10.3.121.31:<0.20108.1>:xdc_replicator:terminate:379] Replication `c85b55a05fc59ffc3bcd724957736be2` (`bucket1/78` -> `http://10.3.121.37:8092/bucket1%2F78/`) failed: {worker_died,<0.20128.1>,
{function_clause,
[{couch_doc,strip_leading_bracket, [<<85,162,187,101,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0>>]},
{couch_doc,to_raw_json_binary,2}
,
{xdc_replicator_worker,'-local_process_batch/6-fun-1-',3},
{lists,foldl,3},
{xdc_replicator_worker,local_process_batch,6},
{xdc_replicator_worker,queue_fetch_loop,5}]}}
[error_logger:error] [2012-07-25 9:20:45] [ns_1@10.3.121.31:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server <0.20108.1> terminating
** Last message in was {'EXIT',<0.20128.1>,
{function_clause,
[{couch_doc,strip_leading_bracket, [<<85,162,187,101,10,0,0,0,0,0,0,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,0,0,0,0>>]},
{couch_doc,to_raw_json_binary,2},
{xdc_replicator_worker,maybe_flush_docs,3}
,
,
{xdc_replicator_worker,local_process_batch,6},
{xdc_replicator_worker,queue_fetch_loop,5}]}}
** When Server state == {rep_state,
{rep,
{"c85b55a05fc59ffc3bcd724957736be2",[]},
<<"bucket1/78">>,
{httpdb,"http://10.3.121.37:8092/bucket1%2F78/",
nil,
[{"Accept","application/json"},
{"User-Agent","CouchDB/1.2.0a-0baf828-git"}],
30000,
[{connect_options,
[{keepalive,true},{nodelay,false}]},
{connect_timeout,30000},
{send_retry,0}],
10,250,nil,20},
[{connection_timeout,30000},
{continuous,false},
{http_connections,20},
{retries,10},
{socket_options,[{keepalive,true},{nodelay,false}]},
{worker_batch_size,500},
{worker_processes,4}],
{user_ctx,null,[<<"_admin">>],undefined},
<<"578f7258f6e8dfec786e35c13d000546">>,78,
xdc_rep_stats_store},
"bucket1/78","http://10.3.121.37:8092/bucket1%2F78/",
{db,<0.4746.0>,<0.4747.0>,nil,
<<"1343232715284354">>,<0.4743.0>,<0.4748.0>,
{db_header,10,70, <<0,0,0,1,125,114,0,0,0,0,16,101,0,0,0,0,66,0,0,0,0, 3,0,0,0,0,71,16>>, <<0,0,0,1,130,44,0,0,0,0,13,91,0,0,0,0,69>>, <<0,0,0,0,240,91,0,0,0,0,0,93>>, 0,nil,nil},
70,
{btree,<0.4743.0>,
{97650,<<0,0,0,0,66,0,0,0,0,3,0,0,0,0,71,16>>,4197},
#Fun<couch_db_updater.7.86200592>,
#Fun<couch_db_updater.8.766827>,
#Fun<couch_btree.5.122018292>,
#Fun<couch_db_updater.9.73139740>,1279,true},
{btree,<0.4743.0>,
{98860,<<0,0,0,0,69>>,3419},
#Fun<couch_db_updater.10.47802347>,
#Fun<couch_db_updater.11.10763047>,
#Fun<couch_db_updater.6.87227503>,
#Fun<couch_db_updater.12.32074001>,1279,true},
{btree,<0.4743.0>,
{61531,<<>>,93},
#Fun<couch_btree.3.114156709>,
#Fun<couch_btree.4.64543194>,
#Fun<couch_btree.5.122018292>,nil,1279,true},
70,<<"bucket1/78">>,
"/opt/couchbase/var/lib/couchbase/data/bucket1/78.couch.1",
[],nil,
{user_ctx,null,[<<"_admin">>],undefined},
nil,
[before_header,after_header,on_file_open],
[]},
{httpdb,"http://10.3.121.37:8092/bucket1%2F78/",nil,
[{"Accept","application/json"},
{"User-Agent","CouchDB/1.2.0a-0baf828-git"}],
30000,
[{connect_options,
[{keepalive,true},{nodelay,false}]},
{connect_timeout,30000},
{send_retry,0}],
10,250,<0.20109.1>,20},
{db,<0.1470.0>,<0.1471.0>,nil,
<<"1343232712948554">>,<0.1464.0>,<0.1472.0>,
{db_header,10,0,nil,nil, <<0,0,0,1,217,160,0,0,0,0,35,196>>, 0,nil,nil},
0,
{btree,<0.1464.0>,nil, #Fun<couch_db_updater.7.86200592>, #Fun<couch_db_updater.8.766827>, #Fun<couch_btree.5.122018292>, #Fun<couch_db_updater.9.73139740>,1279,true},
{btree,<0.1464.0>,nil, #Fun<couch_db_updater.10.47802347>, #Fun<couch_db_updater.11.10763047>, #Fun<couch_db_updater.6.87227503>, #Fun<couch_db_updater.12.32074001>,1279,true},
{btree,<0.1464.0>,
{121248,<<>>,9156},
#Fun<couch_btree.3.114156709>,
#Fun<couch_btree.4.64543194>,
#Fun<couch_btree.5.122018292>,nil,1279,true},
0,<<"bucket1/master">>,
"/opt/couchbase/var/lib/couchbase/data/bucket1/master.couch.1",
[],nil,
{user_ctx,null,[<<"_admin">>],undefined},
nil,
[before_header,after_header,on_file_open],
[create]},
{httpdb,"http://10.3.121.37:8092/bucket1%2Fmaster/",
nil,
[{"Accept","application/json"},
{"User-Agent","CouchDB/1.2.0a-0baf828-git"}],
300000,[],10,250,<0.20117.1>,undefined},
[],
{[{<<"no_changes">>,true}]},
{0,0},
{0,0},
{0,0},
[{1,1},{3,19},{5,70}],
{4,20},
{doc,
<<"_local/78-c85b55a05fc59ffc3bcd724957736be2">>,
{0,<<>>},
{[]},
0,false,[]},
{doc,
<<"_local/78-c85b55a05fc59ffc3bcd724957736be2">>,
{0,<<>>},
{[]},
0,false,[]},
"Wed, 25 Jul 2012 16:20:45 GMT",
<<"1343232715284354">>,<<"1343232702">>,nil,
<0.20124.1>,<0.20126.1>,<0.20125.1>,
[<0.20127.1>,<0.20128.1>,<0.20129.1>],
{rep_stats,18,18,18,18,0},
<<"38dcb722e708f359c73c25fc32aee2ba">>,<0.20123.1>,
nil,#Ref<0.0.7.121009>,nil,#Ref<0.0.7.121010>,nil,70}
** Reason for termination ==
** {worker_died,<0.20128.1>,
{function_clause,
[{couch_doc,strip_leading_bracket, [<<85,162,187,101,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,0,0,0>>]},
{couch_doc,to_raw_json_binary,2},
{xdc_replicator_worker,maybe_flush_docs,3},
{xdc_replicator_worker,'-local_process_batch/6-fun-1-',3},
{lists,foldl,3}
,
,
]}}
[error_logger:error] [2012-07-25 9:20:45] [ns_1@10.3.121.31:error_logger:ale_error_logger_handler:log_report:72]
Attaching logs at https://s3.amazonaws.com/bugdb/jira/xdcr-10/reboot.tar
