[xdcr only]Node crash on Bidirectional replication on Windows 2.0.1

Junyi Xie (Inactive) added a comment - 14/Feb/13 5:53 PM Please upload the logs on both sides, also the memcached dump of the failed node at destination 10.3.2.12

Ketaki Gangal added a comment - 14/Feb/13 6:00 PM Logs here https://s3.amazonaws.com/bugdb/7756/7756.tar

Ketaki Gangal added a comment - 14/Feb/13 6:11 PM Only 2 of the 3 healthy nodes are visible on : view by server option. Added screenshot.

Ketaki Gangal added a comment - 14/Feb/13 6:42 PM Another node 11, now in pending state.

Junyi Xie (Inactive) added a comment - 14/Feb/13 6:43 PM Thanks. Can you please also upload the memcached dump?

Ketaki Gangal added a comment - 14/Feb/13 8:07 PM There is no memcached core on this node. Just found few memcached lost connection - badmatch errors on the node logs btw. debug.5:[user:info,2013-02-14T14:35:15.731, ns_1@10.3.2.12 :ns_memcached-default<0.1313.0>:ns_memcached:terminate:692]Control connection to memcached on ' ns_1@10.3.2.12 ' disconnected: {{badmatch, debug.5- {error, debug.5- timeout}}, debug.5- [{mc_binary, debug.5- quick_stats_recv, debug.5- 3}, debug.5- {mc_binary, debug.5- quick_stats_loop, debug.5- 5}, debug.5- {mc_binary, debug.5- quick_stats,

Junyi Xie (Inactive) added a comment - 15/Feb/13 11:47 AM Ketaki, Thanks for your input. The log tar ball you uploaded does not have the log from 10.3.2.12, which is the crashed node in destination cluster. Junyis-MacBook-Pro:Downloads junyi$ tar xvf 7756.tar x 7756/10.3.121.182-8091-diag.txt.gz x 7756/10.3.121.185-8091-diag.txt.gz x 7756/10.3.2.10-8091-diag.txt.gz x 7756/10.3.2.11-8091-diag.txt.gz x 7756/10.3.2.16-8091-diag.txt.gz x 7756/10.3.2.75-8091-diag.txt.gz x 7756/10.3.2.76-8091-diag.txt.gz x 7756/10.3.2.77-8091-diag.txt.gz Is it possible to include the logs from that crashed node? Thanks.

Ketaki Gangal added a comment - 15/Feb/13 12:36 PM Logs from 10.3.2.12 https://s3.amazonaws.com/bugdb/7757/bug.tar

Junyi Xie (Inactive) added a comment - 15/Feb/13 1:44 PM From logs of 10.3.2.12, there are an avalanche of time-out error in xdcr and ns_server components on that node since 2013-02-14T14:23:30. eventually brought down the node. We have seen such time-out that killed XDCR many times since 2.0GA. I assue the issue would be fixed by the async erlang IO fix in 2.0.1 but seems it is not completely fixed. Also, looks like memcached was not in good shape either. A lot of request from ns-server to memcached takes too long. Not sure it is a bug or due to overloading. ep_engine team, please advise. [ns_server:error,2013-02-15T4:20:50.028, ns_1@10.3.2.12 :<0.16620.0>:ns_memcached:verify_report_long_call:297]call {stats,<<>>} took too long: 1360000 us [ns_server:error,2013-02-15T4:20:50.403, ns_1@10.3.2.12 :<0.16618.0>:ns_memcached:verify_report_long_call:297]call {get_meta,<<"dest_one-cb539de455678520">>,666} took too long: 813000 us [ns_server:error,2013-02-15T4:20:52.106, ns_1@10.3.2.12 :<0.16619.0>:ns_memcached:verify_report_long_call:297]call {get_meta,<<"dest_one-4fe0c5d7aa43671f">>,628} took too long: 532000 us [error_logger:error,2013-02-15T4:20:53.997, ns_1@10.3.2.12 :error_logger<0.6.0>:ale_error_logger_handler:log_msg:76]** Generic server disksup terminating

Jin Lim added a comment - 15/Feb/13 1:51 PM Hi Aliaksey, can you please take a quick look at this issue? Per Junyi's comments above, I also believe there was similar bug tracking ep engine ignoring ns server's requests repeatedly (causing many timeouts on the same request). I am going over the bug list to find that similar bug btw. Thanks!

Junyi Xie (Inactive) added a comment - 15/Feb/13 2:00 PM Hi JIn Which bug are you referring to by "I also believe there was similar bug tracking ep engine ignoring ns server's requests repeatedly (causing many timeouts on the same request)."? Can you please put the bug number here? Thanks.

Jin Lim added a comment - 15/Feb/13 2:38 PM As I mentioned above, I am going over the list to find that bug... stand by please.

Jin Lim added a comment - 15/Feb/13 2:54 PM OK, MB-7571 and MB-7760 both have timeout due to non-responsive memcached. Hope this helps!

Junyi Xie (Inactive) added a comment - 15/Feb/13 3:07 PM Mike, why remove the "couchbase bucket" from the component list? Apparently some operations took too long to finish.

Aliaksey Artamonau added a comment - 15/Feb/13 7:40 PM The system is heavily swapped. So it's not surprising to see those timeouts. Additionally, memcached complains that it could not allocate memory: [couchdb:error,2013-02-15T0:59:17.950, ns_1@10.3.2.12 :<0.5411.9>:couch_log:error:42]Uncaught error in HTTP request: {error,                                  {case_clause,                                   {memcached_error,enomem,undefined}}} Stacktrace: [{capi_replication,is_missing_rev,4},              {capi_replication,'-get_missing_revs/2-fun-0-',4},              {capi_replication,get_missing_revs,2},              {capi_frontend,get_missing_revs,2},              {couch_httpd_db,db_req,2},              {couch_db_frontend,do_db_req,2},              {couch_httpd,handle_request,6},              {mochiweb_http,headers,5}]

Junyi Xie (Inactive) added a comment - 15/Feb/13 7:59 PM - edited Thanks a lot Aliaskey for looking at this. Also thanks JIn for the ep_engine bug Wondering if ep_engine or ns_server folks help me on two questions 1) Is this behavior expected? I mean when system is under heavy workload or in heavily swapped mode, or unable to allocate memory, the node would crash instead of slowing down itself but keeping alive? 2) why memcached was unable to allocate memory, is it a bug or something expected? It is just a bi-directional xdcr on 4:4 nodes cluster, around 1K ops/sec/node, no views, no query. Does not seem to be a very heavy test to me. If bot are expected, that means users are unable to run xdcr with moderate workload on 4:4 windows clusters. :( Thanks.

Junyi Xie (Inactive) added a comment - 15/Feb/13 8:05 PM add "couchbase bucket" into component list, because the question about memory allocation in ep_engine need some attention from ep-engine team.

Aliaksey Artamonau added a comment - 15/Feb/13 8:26 PM I forgot that it was a windows system. I remember that Farshid told me that having a large swap usage is kind of normal and what we should look is hard page faults. Unfortunately we don't have this information in the log file. But I think that enomem errors from memcached is a good evidence that system is really under memory pressure. We know that we don't behave good in such conditions. Besides that, as far as I remember we've seen timeouts on windows even after we mostly fixed them on linux. Sriram was investigating this. But I don't know current status of this.

Jin Lim added a comment - 16/Feb/13 1:24 AM Thanks Aliaksey and Junyi for looking into this issue and providing vital info. I added Mike for Junyi's ep engine question.

Mike Wiederhold added a comment - 16/Feb/13 1:17 PM I originally took the couchbase bucket component off of this issue because historically crashes due to timeouts from ns_server were "caused" by ns_server. What I mean by that is that ns_server will abort whatever it's doing if the a timeout hits since it decides memcached is unresponsive. I'm aware that there's more to this problem and that memcached not responding in 30 seconds isn't great either, but there also is no crash on the memcached side so I'm not sure how I could have debugged the issue. In any case I need to discuss this timeout issue further with Alk. 2) There are two reasons memcached wouldn't be able to allocate memory. The first is if system runs out of memory in which case we would see memcached crash with sigkill from the os. This is possible if not enough swap space is allocated. It also, doesn't seem like what happened here. The second reason is that meta data has filled the entire space allocated for the cache. In this case memcached would return enomem for any operations sent to it.

Junyi Xie (Inactive) added a comment - 18/Feb/13 11:06 AM From ns_server and ep_engine folks, the node crash in ns_server and ep_engine is somehow "expected". I do not know if anything I can do from XDCR perspective.

Jin Lim added a comment - 18/Feb/13 11:35 AM Based on Mike's comments above and Alk's debugging that memcachd complained about enomem for requests being sent: [couchdb:error,2013-02-15T0:59:17.950, ns_1@10.3.2.12 :<0.5411.9>:couch_log:error:42]Uncaught error in HTTP request: {error,{case_clause,{memcached_error,enomem,undefined}}} we may want to rerun the teest with more ram or have one of our engineers take a look at the live cluster still showing the node crash. QE please advise on this. Thanks.

Aliaksey Artamonau added a comment - 18/Feb/13 12:24 PM CC Siri Siri, can you suggest what QE should be looking at while running the test to discriminate between memory pressure and other causes of the timeouts. And it would be even better if we could collect this information in ns_server.

Farshid Ghods (Inactive) added a comment - 19/Feb/13 11:24 AM QE is going to rerun this test on EC2 cluster and see if this issue is reproducible there as well

Farshid Ghods (Inactive) added a comment - 21/Feb/13 2:43 PM unable to reproduce on ec2