[XDCR] core dumps found in xdcr jobs

The following test cases passed but have been failed on Jenkins due to core dumps found:

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=ALL -t xdcr.biXDCR.bidirectional.load_with_async_ops_and_joint_sets,items=10000,ctopology=chain,rdirection=bidirection,sasl_buckets=1,replication_type=xmem,GROUP=P0;xmem (core1.zip) - https://s3.amazonaws.com/bugdb/jira/MB-18779/logs.tar

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=unidirection,ctopology=chain,update=C1,delete=C1,rebalance=C2,num_rebalance=2,GROUP=P0;xmem (core2.zip) - https://s3.amazonaws.com/bugdb/jira/MB-18779/logs01.tar

2016-03-15 11:21:46 | INFO | MainProcess | test_thread | [xdcrnewbasetests.__initialize_error_count_dict] {'172.23.106.115': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.162': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.163': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.161': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.164': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.161': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.106.251': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.172': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}}

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_out_master,items=100000,rdirection=bidirection,ctopology=chain,update=C1-C2,delete=C1-C2,rebalance=C2,GROUP=P1 (core3.zip)

• https://s3.amazonaws.com/bugdb/jira/MB-18779/logs2.tar

  [2016-03-15 17:17:23,827] - [xdcrnewbasetests:2457] INFO - {'172.23.106.115': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.162': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.163': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.161': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.121.164': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.107.161': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.106.251': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}, '172.23.105.172': {'non-recoverable error from xmem client. response status=KEY_ENOENT': 0, 'panic:': 0}}

./testrunner i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.swap_rebalance_replication_with_view_queries_and_ops,items=10000,update=C1,delete=C1,rdirection=unidirection,rebalance=C2,poll_timeout=180,GROUP=P0;xmem (core4.zip) https://s3.amazonaws.com/bugdb/jira/MB-18779/logs3.tar

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,fail_on_errors=1,GROUP=P0 -t xdcr.xdcrTopologyTests.XDCRTopologyTest.load_with_failover,items=10000,chain_length=3,rdirection=unidirection,ctopology=star,rebalance=C1,num_rebalance=1,replication_type=xmem,GROUP=P0 (core5.zip) - https://s3.amazonaws.com/bugdb/jira/MB-18779/test5.zip

./testrunner -i INI_FILE.ini get-cbcollect-info=True,get-coredumps=True,get-logs=False,stop-on-failure=False,fail_on_errors=1,GROUP=CHAIN -t xdcr.uniXDCR.unidirectional.load_with_async_ops,items=100000,rdirection=unidirection,ctopology=chain,expires=60,standard_buckets=1,sasl_buckets=2,default_bucket=False,delete=C1,GROUP=CHAIN;P1 (core6.zip) - https://s3.amazonaws.com/bugdb/jira/MB-18779/test6.zip

./testrunner -i INI_FILE get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,demand_encryption=1,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.async_rebalance_in,items=100000,rdirection=bidirection,ctopology=chain,update=C1-C2,delete=C1-C2,expires=240,rebalance=C2,num_rebalance=1,GROUP=P1 (core7.zip) - https://s3.amazonaws.com/bugdb/jira/MB-18779/test7.zip

./testrunner -i INI_FILE get-cbcollect-info=True,get-logs=False,stop-on-failure=False,get-coredumps=True,demand_encryption=1,fail_on_errors=1,GROUP=ALL -t xdcr.rebalanceXDCR.Rebalance.swap_rebalance_replication_with_ddoc_compaction,items=100000,rdirection=bidirection,rebalance=C2,GROUP=P1,poll_timeout=900 (core8.zip) - https://s3.amazonaws.com/bugdb/jira/MB-18779/test8.zip

Please investigate these core dumps - if needed we can create separate tickets to track each one of them