{UPR} :: Rebalance hanging during Views + Rebalance tests

Jenkins Ref Link:
http://qa.sc.couchbase.com/job/centos_x64--29_01--new_view_all-P1/56/console

Tests to Reproduce:
./testrunner -i <yourfile>.ini get-cbcollect-info=True,get-logs=True, -t view.createdeleteview.CreateDeleteViewTests.ddoc_ops_during_failover,ddoc_ops=create,test_with_view=True,num_ddocs=3,num_views_per_ddoc=2,items=200000,nodes_out=2,replicas=2

Tests Failed:
ddoc_ops_during_failover,ddoc_ops=update,test_with_view=True,num_ddocs=2,num_views_per_ddoc=2,items=200000,nodes_out=1,sasl_buckets=1,standard_buckets=1
ddoc_ops_during_failover,ddoc_ops=delete,test_with_view=True,num_ddocs=2,num_views_per_ddoc=3,items=200000,nodes_out=2,replicas=2
ddoc_ops_during_failover,ddoc_ops=create,test_with_view=False,num_ddocs=3,num_views_per_ddoc=2,items=200000,nodes_out=2,replicas=2,default_bucket=False,sasl_buckets=1,standard_buckets=1
ddoc_ops_during_failover,ddoc_ops=update,test_with_view=False,num_ddocs=2,num_views_per_ddoc=2,items=200000,nodes_out=2,replicas=2
ddoc_ops_during_failover,ddoc_ops=delete,test_with_view=False,num_ddocs=2,num_views_per_ddoc=3,items=200000,nodes_out=1,sasl_buckets=1,standard_buckets=1

Logs:
[error_logger:error,2014-05-06T2:17:20.301,ns_1@172.23.107.20:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]
=========================CRASH REPORT=========================
crasher:
initial call: ns_single_vbucket_mover:mover/6
pid: <0.20483.67>
registered_name: []
exception exit: {unexpected_exit,{'EXIT',<0.19575.67>,shutdown}}
in function ns_single_vbucket_mover:spawn_and_wait/1
in call from ns_single_vbucket_mover:wait_master_seqno_persisted_on_replicas/5
in call from ns_single_vbucket_mover:mover_inner_upr/6
in call from misc:try_with_maybe_ignorant_after/2
in call from ns_single_vbucket_mover:mover/6
ancestors: [<0.19575.67>,<0.19503.67>,<0.13868.0>,mb_master_sup,
mb_master,ns_server_sup,'ns_server_sup-wrapper',
ns_server_cluster_sup,<0.58.0>]
messages: [

{'EXIT',<0.19575.67>,shutdown}

]
links: [<0.19575.67>]
dictionary: [

{cleanup_list,[<0.20522.67>]}

]
trap_exit: true
status: running
heap_size: 2584
stack_size: 24
reductions: 1646
neighbours:
[ns_server:debug,2014-05-06T2:17:20.302,ns_1@172.23.107.20:upr_replicator-default-ns_1@172.23.107.21<0.14657.66>:upr_replicator:terminate:100]Terminating with reason nuke. Nuked connection "ns_server:ns_1@172.23.107.21->ns_1@172.23.107.20:default" with result [ok, ok].
[error_logger:error,2014-05-06T2:17:20.303,ns_1@172.23.107.20:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]

=========================CRASH REPORT=========================
crasher:
initial call: erlang:apply/2
pid: <0.19503.67>
registered_name: []
exception exit: stopped
in function ns_rebalancer:run_mover/7
in call from ns_rebalancer:rebalance/6
in call from ns_rebalancer:'rebalance/5-fun-3'/6
in call from lists:foreach/2
in call from ns_rebalancer:rebalance/5
ancestors: [<0.13868.0>,mb_master_sup,mb_master,ns_server_sup,
'ns_server_sup-wrapper',ns_server_cluster_sup,<0.58.0>]
messages: []
links: [<0.19524.67>,<0.13868.0>]
dictionary: [{random_seed,{3688,3451,10969}}]
trap_exit: false
status: running
heap_size: 75025
stack_size: 24
reductions: 2605712
neighbours:
neighbour: [

{pid,<0.19526.67>}

,

{registered_name,[]},
{initial_call,{erlang,apply,['Argument__1','Argument__2']}},
{current_function,
{ns_rebalance_observer,docs_left_updater_loop,1}},
{ancestors, [<0.19524.67>,<0.19503.67>,<0.13868.0>,mb_master_sup, mb_master,ns_server_sup,'ns_server_sup-wrapper', ns_server_cluster_sup,<0.58.0>]},
{messages,[]},
{links,[<0.19524.67>,<0.309.0>]},
{dictionary,[]},
{trap_exit,false},
{status,waiting},
{heap_size,46368},
{stack_size,6},
{reductions,2381627}]
neighbour: [{pid,<0.19524.67>},
{registered_name,[]}

,
{initial_call,{ns_rebalance_observer,init,['Argument__1']}},
{current_function,{gen_server,loop,6}},

{ancestors,[<0.19503.67>,<0.13868.0>,mb_master_sup, mb_master,ns_server_sup,'ns_server_sup-wrapper', ns_server_cluster_sup,<0.58.0>]}

,

{messages,[]}

,

{links,[<0.19525.67>,<0.19526.67>,<0.19503.67>]}

,

{dictionary,[]}

,

{trap_exit,false}

,

{status,waiting}

,

{heap_size,121393}

,

{stack_size,9}

,

{reductions,1033492}

]
error_logger:error,2014-05-06T2:17:20.304,ns_1@172.23.107.20:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]
=========================CRASH REPORT=========================
crasher:
initial call: upr_proxy:init/1
pid: <0.14658.66>
registered_name: 'upr_consumer_conn-default-ns_1@172.23.107.21'
exception exit: nuke
in function gen_server:terminate/6
ancestors: ['upr_replicator-default-ns_1@172.23.107.21',
'upr_sup-default','single_bucket_sup-default',<0.2468.66>]
messages: []
links: [<0.14657.66>]
dictionary: []
trap_exit: true
status: running
heap_size: 4181
stack_size: 24
reductions: 48952448
neighbours:

[error_logger:error,2014-05-06T2:17:20.304,ns_1@172.23.107.20:error_logger<0.6.0>:ale_error_logger_handler:do_log:207]** Generic server <0.14659.66> terminating

• • Last message in was {'EXIT',<0.14657.66>,nuke}
  • When Server state ==
    Unknown macro: {state,#Port<0.974982>, {producer, "ns_server:ns_1@172.23.107.21->ns_1@172.23.107.20:default", 'ns_1@172.23.107.21',"default"}, <<>>,upr_producer_conn,[],#Port<0.974981>, <0.14658.66>}
  • Reason for termination ==
  • nuke
    [ns_server:debug,2014-05-06T2:17:20.549,ns_1@172.23.107.20:janitor_agent-default<0.2490.66>:janitor_agent:set_rebalance_mref:872]Killing rebalance-related subprocess: <0.22155.67>
    [ns_server:debug,2014-05-06T2:17:20.549,ns_1@172.23.107.20:janitor_agent-default<0.2490.66>:janitor_agent:set_rebalance_mref:872]Killing rebalance-related subprocess: <0.22099.67>
    [ns_server:debug,2014-05-06T2:17:20.549,ns_1@172.23.107.20:janitor_agent-default<0.2490.66>:janitor_agent:set_rebalance_mref:872]Killing rebalance-related subprocess: <0.22057.67>
    [ns_server:debug,2014-05-06T2:17:20.549,ns_1@172.23.107.20:janitor_agent-default<0.2490.66>:janitor_agent:set_rebalance_mref:872]Killing rebalance-related subprocess: <0.22020.67>

Uploading Logs