Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-34824

Intermittent Flush Bucket Failures

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      We have a regression test that performs a flush bucket operation and recently we started seeing some flush operations failures. The test uses 5 cluster_run nodes and when the flush was attempted, the following nodes were in the cluster (0, 2, 3, 4). We re-try the flush operation upto 5 times with one second wait time between them in case of a flush failure. All 5 attempts failed in the attached logs.

      ns_server error log shows:

      [ns_server:error,2019-07-01T18:53:10.428-07:00,n_0@127.0.0.1:<0.24310.0>:ns_config_rep:synchronize_remote:296]Failed to synchronize config to some nodes: [{'n_4@127.0.0.1', {exit, {noproc, {gen_server,call, [{ns_config_rep,'n_4@127.0.0.1'}, synchronize_everything,infinity]}}}}] [ns_server:error,2019-07-01T18:53:10.429-07:00,n_0@127.0.0.1:<0.23964.0>:menelaus_web:loop:163]Server error during processing: ["web request failed", {path, "/pools/default/buckets/gbook_users/controller/doFlush"}, {method,'POST'}, {type,exit}, {what, {{{badmatch, {error, [{'n_4@127.0.0.1', {exit, {noproc, {gen_server,call, [{ns_config_rep,'n_4@127.0.0.1'}, synchronize_everything, infinity]}}}}]}}, [{ns_janitor,config_sync,3, [{file,"src/ns_janitor.erl"}, {line,223}]}, {ns_janitor,cleanup_with_states,5, [{file,"src/ns_janitor.erl"}, {line,129}]}, {ns_janitor, '-cleanup_membase_bucket/4-fun-0-',4, [{file,"src/ns_janitor.erl"},{line,58}]}, {async,'-async_init/4-fun-2-',3, [{file,"src/async.erl"},{line,211}]}]}, {gen_statem,call, [{via,leader_registry,ns_orchestrator}, {flush_bucket,"gbook_users"}, infinity]}}}, {trace, [{gen,do_call,4, [{file,"gen.erl"},{line,177}]}, {gen,do_for_proc,2, [{file,"gen.erl"},{line,271}]}, {gen_statem,call_dirty,4, [{file,"gen_statem.erl"},{line,591}]}, {menelaus_web_buckets, do_handle_bucket_flush,2, [{file,"src/menelaus_web_buckets.erl"}, {line,775}]}, {request_throttler,do_request,3, [{file,"src/request_throttler.erl"}, {line,59}]}, {menelaus_web,loop,2, [{file,"src/menelaus_web.erl"}, {line,141}]}, {mochiweb_http,headers,5, [{file, "/home/couchbase/jenkins/workspace/cbas-cbcluster-test2/couchdb/src/mochiweb/mochiweb_http.erl"}, {line,94}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,247}]}]}]
      

      Please let us know if this is expected and we need to wait for a period of time before attempting the flush again.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              timofey.barmin Timofey Barmin
              murtadha.hubail Murtadha Hubail
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty