Magma rebalance tests fail with reason mover_crashed on 7.0.0-1722 and 1006.5.1-1113

build: 7.0.0-1722

Job (Rebalance-in, 4 -> 5): http://perf.jenkins.couchbase.com/job/titan-reb/1194/

Log:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1194/172.23.96.100.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1194/172.23.96.101.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1194/172.23.96.102.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1194/172.23.96.103.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1194/172.23.96.104.zip

 

Job (Rebalance-swap, 4 -> 4): http://perf.jenkins.couchbase.com/job/titan-reb/1195/

Log: 

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1195/172.23.96.100.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1195/172.23.96.101.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1195/172.23.96.102.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1195/172.23.96.103.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1195/172.23.96.104.zip

 

Job (Rebalance-out, 5 -> 4): http://perf.jenkins.couchbase.com/job/titan-reb/1196/

Log:

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1196/172.23.96.100.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1196/172.23.96.101.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1196/172.23.96.102.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1196/172.23.96.103.zip

https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-1196/172.23.96.104.zip

 

Rebalance exited with reason {mover_crashed,

{unexpected_exit,

{'EXIT',<0.26364.21>,

{{wait_seqno_persisted_failed,"bucket-1",166,

979600,

[{'ns_1@172.23.96.102',

{'EXIT',

{badmatch,{error,closed,

[

{dcp_proxy,handle_packet,2, [\{file,"src/dcp_proxy.erl"}

,

{line,189}

]},

{dcp_proxy,process_data_loop,3, [\{file,"src/dcp_proxy.erl"}

,

{line,368}

]},

{dcp_proxy,handle_info,2, [\{file,"src/dcp_proxy.erl"}

,

{line,103}

]},

{gen_server,try_dispatch,4, [\{file,"gen_server.erl"}

,{line,616}]},

{gen_server,handle_msg,6, [\{file,"gen_server.erl"}

,{line,686}]},

{proc_lib,init_p_do_apply,3, [\{file,"proc_lib.erl"}

,{line,247}]}]},

{gen_server,call,

[

{'janitor_agent-bucket-1', 'ns_1@172.23.96.102'}

,

{if_rebalance,<0.19525.21>,

{wait_seqno_persisted,166,979600}},

infinity]}}}}]},

[

{ns_single_vbucket_mover, '-wait_seqno_persisted_many/5-fun-2-',5, [\{file,"src/ns_single_vbucket_mover.erl"}

,

{line,488}

]},

{proc_lib,init_p,3, [\{file,"proc_lib.erl"}

,{line,232}]}]}}}}.

Rebalance Operation Id = a104b7640ae61f0d0d7f480f6dcbefa9