Ephemeral: memcached crashes during initial XDC replication

2017-06-03T20:44:23.483883-07:00 FATAL Breakpad caught crash in memcached version 5.0.0-3008. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/467ae355-ecac-dad7-44057dd5-2ae8d77f.dmp before terminating.

2017-06-03T20:44:23.483916-07:00 FATAL Stack backtrace of crashed thread:

2017-06-03T20:44:23.495424-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x34cc5]

2017-06-03T20:44:23.495488-07:00 FATAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3d4) [0x400000+0x878b4]

2017-06-03T20:44:23.495526-07:00 FATAL     /opt/couchbase/bin/memcached(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0x97) [0x400000+0x87bc7]

2017-06-03T20:44:23.495564-07:00 FATAL     /lib64/libpthread.so.0() [0x7f3022709000+0xf100]

2017-06-03T20:44:23.495650-07:00 FATAL     /lib64/libc.so.6(gsignal+0x37) [0x7f30207c1000+0x355f7]

2017-06-03T20:44:23.495714-07:00 FATAL     /lib64/libc.so.6(abort+0x148) [0x7f30207c1000+0x36ce8]

2017-06-03T20:44:23.495800-07:00 FATAL     /lib64/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x165) [0x7f302109b000+0x609d5]

2017-06-03T20:44:23.495834-07:00 FATAL     /opt/couchbase/bin/memcached() [0x400000+0x3609a]

2017-06-03T20:44:23.495903-07:00 FATAL     /lib64/libstdc++.so.6() [0x7f302109b000+0x5e946]

2017-06-03T20:44:23.495970-07:00 FATAL     /lib64/libstdc++.so.6() [0x7f302109b000+0x5e973]

2017-06-03T20:44:23.496036-07:00 FATAL     /lib64/libstdc++.so.6() [0x7f302109b000+0x5eb93]

2017-06-03T20:44:23.496070-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0xcf983]

2017-06-03T20:44:23.496095-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0xd1b6d]

2017-06-03T20:44:23.496122-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0x43038]

2017-06-03T20:44:23.496146-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0x432a0]

2017-06-03T20:44:23.496169-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0x43868]

2017-06-03T20:44:23.496192-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0x3e703]

2017-06-03T20:44:23.496215-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0x3ea44]

2017-06-03T20:44:23.496248-07:00 FATAL     /opt/couchbase/bin/../lib/../lib/ep.so() [0x7f301c03e000+0x9e40d]

2017-06-03T20:44:23.496277-07:00 FATAL     /opt/couchbase/bin/../lib/libplatform_so.so.0.1.0() [0x7f3022d46000+0x6bce]

2017-06-03T20:44:23.496307-07:00 FATAL     /lib64/libpthread.so.0() [0x7f3022709000+0x7dc5]

2017-06-03T20:44:23.496389-07:00 FATAL     /lib64/libc.so.6(clone+0x6d) [0x7f30207c1000+0xf6ced]

(gdb) backtrace

#0  0x00007f30207f65f7 in raise () from /usr/lib64/libc-2.17.so

#1  0x00007f30207f7ce8 in abort () from /usr/lib64/libc-2.17.so

#2  0x00007f30210fb9d5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6.0.19

#3  0x000000000043609a in backtrace_terminate_handler () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:103

#4  0x00007f30210f9946 in ?? () from /usr/lib64/libstdc++.so.6.0.19

#5  0x00007f30210f9973 in std::terminate() () from /usr/lib64/libstdc++.so.6.0.19

#6  0x00007f30210f9b93 in __cxa_throw () from /usr/lib64/libstdc++.so.6.0.19

#7  0x00007f301c10d983 in setBegin (start=<optimized out>, this=0x7f2f884ba1c8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/linked_list.h:97

#8  BasicLinkedList::RangeIteratorLL::operator++ (this=<optimized out>) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/linked_list.cc:526

#9  0x00007f301c10fb6d in SequenceList::RangeIterator::operator++ (this=this@entry=0x7f2f883ef5b8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/seqlist.cc:31

#10 0x00007f301c081038 in DCPBackfillMemoryBuffered::scan (this=this@entry=0x7f2f883ef580) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_memory.cc:241

#11 0x00007f301c0812a0 in DCPBackfillMemoryBuffered::create (this=this@entry=0x7f2f883ef580, evb=...)

    at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_memory.cc:195

#12 0x00007f301c081868 in DCPBackfillMemoryBuffered::run (this=0x7f2f883ef580) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill_memory.cc:133

#13 0x00007f301c07c703 in BackfillManager::backfill (this=0x7f2f7c472e60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:297

#14 0x00007f301c07ca44 in BackfillManagerTask::run (this=0x7f11f248bb40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/dcp/backfill-manager.cc:71

#15 0x00007f301c0dc40d in ExecutorThread::run (this=0x7f2ff0145c00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/executorthread.cc:120

#16 0x00007f3022d4cbce in run (this=0x7f2ff01957a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:58

#17 platform_thread_wrap (arg=0x7f2ff01957a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:71

#18 0x00007f3022710dc5 in start_thread () from /usr/lib64/libpthread-2.17.so

#19 0x00007f30208b7ced in clone () from /usr/lib64/libc-2.17.so

2017-06-04T08:58:28.193-07:00 ERRO GOXDCR.PipelineMgr: Failed to construct a new pipeline with topic d418c2751ff05b97af9a5355793b9104/bucket-1/bucket-1: Invalid configuration. No source nozzle can be constructed since the source kv nodes are not the master for any vbuckets.