XDCR checkpointing : ep-engine does not generate new failover log after remote node crash

Scenario
-------------

• XDCR over UPR (uni-dir replication between two 1-node clusters
• Internal replication uses TAP
• Add some mutations to vb449. Check remote end vb_uuid and high_seqno.
• Kill couchbase processes at remote end.
• Start couchbase
• Check remote vb_uuid and high_seqno. It remains unchanged.

XDCR checkpointing is designed to fail (based on a new failover log after a remote node crash) indicative of possible data loss. Since failover id does not change(because of this bug), source continues to replicate and checkpoint not knowing about the crash.

[root@centos-64-x64 logs]# /opt/couchbase/bin/cbstats 10.3.4.188:11210 vbucket-seqno 449
vb_449:high_seqno: 38
vb_449:purge_seqno: 0
vb_449:uuid: 156839137804859

[root@centos-64-x64 logs]# ps -aux|grep couchbase
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
498 10095 0.0 0.0 10824 488 ? S 16:42 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon
498 10117 0.1 0.5 1223412 23180 ? Sl 16:42 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -hidden -name babysitter_of_ns_1@127.0.0.1 -setcookie nocookie -noshell -noinput -noshell -noinput -run ns_babysitter_bootstrap – -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini -ns_babysitter cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie" -ns_server config_path "/opt/couchbase/etc/couchbase/static_config" -ns_server pidfile "/opt/couchbase/var/lib/couchbase/couchbase-server.pid" -ns_server cookiefile "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie-ns-server" -ns_server enable_mlockall false
498 10144 6.0 3.2 1512420 130488 ? Ssl 16:42 0:36 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K true -MMmcs 30 – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -setcookie nocookie -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21299 error_logger false -sasl sasl_error_logger false -nouser -run child_erlang child_start ns_bootstrap – -smp enable -couch_ini /opt/couchbase/etc/couchdb/default.ini /opt/couchbase/etc/couchdb/default.d/capi.ini /opt/couchbase/etc/couchdb/default.d/geocouch.ini /opt/couchbase/etc/couchdb/local.ini
498 10175 0.0 0.0 4056 592 ? Ss 16:42 0:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/memsup
498 10176 0.0 0.0 4052 444 ? Ss 16:42 0:00 /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
498 10182 0.2 0.0 794052 2252 ? Ssl 16:42 0:01 /opt/couchbase/lib/ns_server/erlang/lib/ns_server/priv/i386-linux-godu
498 10186 0.2 0.0 39012 2132 ? Ss 16:42 0:01 /opt/couchbase/lib/erlang/lib/ssl-4.1.6/priv/bin/ssl_esock
498 10187 0.7 0.0 172276 3424 ? Ssl 16:42 0:04 /opt/couchbase/bin/moxi -Z port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200 -z url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming -p 0 -Y y -O stderr
498 10188 18.5 2.4 350388 99016 ? Ssl 16:42 1:51 /opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcached.json
498 10189 0.1 0.6 174168 24928 ? Ssl 16:42 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -P 327680 -K true – -root /opt/couchbase/lib/erlang -progname erl – -home /opt/couchbase – -smp enable -kernel error_logger false -sasl sasl_error_logger false -nouser -pa . /opt/couchbase/lib/erlang/lib/kernel-2.14.5/ebin /opt/couchbase/lib/erlang/lib/stdlib-1.17.5/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_ssl_proxy/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_server/ebin /opt/couchbase/lib/ns_server/erlang/lib/ns_babysitter/ebin /opt/couchbase/lib/ns_server/erlang/lib/mlockall/ebin /opt/couchbase/lib/ns_server/erlang/lib/gen_smtp/ebin /opt/couchbase/lib/ns_server/erlang/lib/erlwsh/ebin /opt/couchbase/lib/ns_server/erlang/lib/ale/ebin /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/ebin /opt/couchbase/lib/couchdb/erlang/lib/mochiweb-1.4.1/ebin /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/lhttpc-1.3/ebin /opt/couchbase/lib/couchdb/erlang/lib/etap/ebin /opt/couchbase/lib/couchdb/erlang/lib/erlang-oauth/ebin /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_upr-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_set_view-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch_index_merger-1.0.0/ebin /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-961ad59-git/ebin /opt/couchbase/lib/couchdb/plugins/geocouch/ebin /opt/couchbase/lib/erlang/lib/xmerl-1.2.10/ebin /opt/couchbase/lib/erlang/lib/webtool-0.8.9/ebin /opt/couchbase/lib/erlang/lib/typer-0.9.2/ebin /opt/couchbase/lib/erlang/lib/tv-2.1.4.7/ebin /opt/couchbase/lib/erlang/lib/tools-2.6.6.5/ebin /opt/couchbase/lib/erlang/lib/toolbar-1.4.2/ebin /opt/couchbase/lib/erlang/lib/test_server-3.4.5/ebin /opt/couchbase/lib/erlang/lib/syntax_tools-1.6.7.1/ebin /opt/couchbase/lib/erlang/lib/ssl-4.1.6/ebin /opt/couchbase/lib/erlang/lib/ssh-2.0.8/ebin /opt/couchbase/lib/erlang/lib/snmp-4.21.1/ebin /opt/couchbase/lib/erlang/lib/sasl-2.1.10/ebin /opt/couchbase/lib/erlang/lib/runtime_tools-1.8.6/ebin /opt/couchbase/lib/erlang/lib/reltool-0.5.7/ebin /opt/couchbase/lib/erlang/lib/public_key-0.13/ebin /opt/couchbase/lib/erlang/lib/pman-2.7.1/ebin /opt/couchbase/lib/erlang/lib/percept-0.8.6/ebin /opt/couchbase/lib/erlang/lib/parsetools-2.0.6/ebin /opt/couchbase/lib/erlang/lib/otp_mibs-1.0.6/ebin /opt/couchbase/lib/erlang/lib/os_mon-2.2.7/ebin /opt/couchbase/lib/erlang/lib/orber-3.6.22/ebin /opt/couchbase/lib/erlang/lib/observer-0.9.10/ebin /opt/couchbase/lib/erlang/lib/mnesia-4.5/ebin /opt/couchbase/lib/erlang/lib/inviso-0.6.2/ebin /opt/couchbase/lib/erlang/lib/inets-5.7.1/ebin /opt/couchbase/lib/erlang/lib/ic-4.2.27/ebin /opt/couchbase/lib/erlang/lib/hipe-3.8.1/ebin /opt/couchbase/lib/erlang/lib/gs-1.5.14/ebin /opt/couchbase/lib/erlang/lib/eunit-2.2.1/ebin /opt/couchbase/lib/erlang/lib/et-1.4.4/ebin /opt/couchbase/lib/erlang/lib/erts-5.8.5/ebin /opt/couchbase/lib/erlang/lib/erl_interface-3.7.5 /opt/couchbase/lib/erlang/lib/erl_docgen-0.2.6/ebin /opt/couchbase/lib/erlang/lib/edoc-0.7.9/ebin /opt/couchbase/lib/erlang/lib/docbuilder-0.9.8.11/ebin /opt/couchbase/lib/erlang/lib/diameter-0.10/ebin /opt/couchbase/lib/erlang/lib/dialyzer-2.4.4/ebin /opt/couchbase/lib/erlang/lib/debugger-3.2.6/ebin /opt/couchbase/lib/erlang/lib/crypto-2.0.4/ebin /opt/couchbase/lib/erlang/lib/cosTransactions-1.2.11/ebin /opt/couchbase/lib/erlang/lib/cosTime-1.1.11/ebin /opt/couchbase/lib/erlang/lib/cosProperty-1.1.14/ebin /opt/couchbase/lib/erlang/lib/cosNotification-1.1.17/ebin /opt/couchbase/lib/erlang/lib/cosFileTransfer-1.1.12/ebin /opt/couchbase/lib/erlang/lib/cosEventDomain-1.1.11/ebin /opt/couchbase/lib/erlang/lib/cosEvent-2.1.11/ebin /opt/couchbase/lib/erlang/lib/compiler-4.7.5/ebin /opt/couchbase/lib/erlang/lib/common_test-1.5.5/ebin /opt/couchbase/lib/erlang/lib/asn1-1.6.18/ebin /opt/couchbase/lib/erlang/lib/appmon-2.1.13/ebin -run child_erlang child_start ns_ssl_proxy
root 10676 0.0 0.0 103240 880 pts/0 S+ 16:52 0:00 grep couchbase

[root@centos-64-x64 logs]# kill -9 10117

[root@centos-64-x64 logs]# ps -aux|grep couchbase
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
498 10095 0.0 0.0 10824 488 ? S 16:42 0:00 /opt/couchbase/lib/erlang/erts-5.8.5/bin/epmd -daemon
root 10678 0.0 0.0 103236 864 pts/0 S+ 16:53 0:00 grep couchbase

[root@centos-64-x64 logs]# kill -9 10095

[root@centos-64-x64 logs]# ps -aux|grep couchbase
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root 10717 0.0 0.0 103236 864 pts/0 S+ 16:53 0:00 grep couchbase

[root@centos-64-x64 logs]# service couchbase-server start
Starting couchbase-server [ OK ]

[root@centos-64-x64 logs]# /opt/couchbase/bin/cbstats 10.3.4.188:11210 vbucket-seqno 449
vb_449:high_seqno: 38
vb_449:purge_seqno: 0
vb_449:uuid: 156839137804859