Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7286

UI logs showing multiple errors, (XDCR bi and uni, at the start of a rebalance-in operation): Server error during processing: ["web request failed" ...

    Details

      Description

      + 5 nodes rebalance in on each cluster
      Cluster setup: c1:c2::10:10
      biXDCR_bucket: c1 <---> c2
      uniXDCR_src: c1 ---> c2 :uniXDCR_dest
      Front end loads on c1 and c2 for biXDCR_bucket, and on c1 for uniXDCR_src.
      c1: http://ec2-177-71-230-72.sa-east-1.compute.amazonaws.com:8091/
      c2: http://ec2-175-41-186-167.ap-southeast-1.compute.amazonaws.com:8091/

      UI reports whole bunch of these errors at the start of rebalancing operation and with the front end loads:
      Server error during processing: ["web request failed",

      {path,"/pools/default"}

      ,

      {type,exit}

      ,
      {what,
      {timeout,

      {gen_server,call, [ns_doctor,get_tasks_version]}

      }},
      {trace,
      [

      {gen_server,call,2}

      ,

      {menelaus_web,build_pool_info,4}

      ,

      {menelaus_web,handle_pool_info_wait,5}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}] (repeated 1 times)

      Will attach grabbed diags from the particular server in a bit.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        abhinav Abhinav Dangeti added a comment -

        UI reports that this error seems to be on server node: ns_1@ec2-177-71-230-72.sa-east-1.compute.amazonaws.com
        Grabbed diags: https://s3.amazonaws.com/bugdb/MB-7286/ec2-177-71-230-72.sa-east-1.compute.amazonaws.com-8091-diag.txt.gz

        Show
        abhinav Abhinav Dangeti added a comment - UI reports that this error seems to be on server node: ns_1@ec2-177-71-230-72.sa-east-1.compute.amazonaws.com Grabbed diags: https://s3.amazonaws.com/bugdb/MB-7286/ec2-177-71-230-72.sa-east-1.compute.amazonaws.com-8091-diag.txt.gz
        Hide
        steve Steve Yen added a comment -

        from bug-scrub.

        ketaki: errors seem to go away after 10 minutes, but cluster is unaccessible?

        Show
        steve Steve Yen added a comment - from bug-scrub. ketaki: errors seem to go away after 10 minutes, but cluster is unaccessible?
        Hide
        steve Steve Yen added a comment -

        per-bug-scrub, moved to 2.0.1

        Show
        steve Steve Yen added a comment - per-bug-scrub, moved to 2.0.1
        Hide
        kzeller kzeller added a comment -

        Added to RN:

        During a rebalance operation for clusters undergoing uni- and bi-directional replication
        via XDCR, the following server errors may appear, which are currently under
        investigation:

        Show
        kzeller kzeller added a comment - Added to RN: During a rebalance operation for clusters undergoing uni- and bi-directional replication via XDCR, the following server errors may appear, which are currently under investigation:
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        Seems to me these errors usually mean the system is busy working on something which is pretty heavy (like rebalance, etc), and is unable to respond to UI request timely. Waiting for triage from ns_server team.

        Show
        junyi Junyi Xie (Inactive) added a comment - Seems to me these errors usually mean the system is busy working on something which is pretty heavy (like rebalance, etc), and is unable to respond to UI request timely. Waiting for triage from ns_server team.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )

        Show
        farshid Farshid Ghods (Inactive) added a comment - deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        This logs are insufficient.

        Please reproduce on 2.1.0

        5 <-> 5 XDCR is IMHO insane

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - This logs are insufficient. Please reproduce on 2.1.0 5 <-> 5 XDCR is IMHO insane
        Hide
        abhinav Abhinav Dangeti added a comment -

        Close for now, will reopen with additional logs if issue seen again.

        Show
        abhinav Abhinav Dangeti added a comment - Close for now, will reopen with additional logs if issue seen again.
        Hide
        venu Venu Uppalapati (Inactive) added a comment -

        Aruna, this is XDCR issue that was incorrectly assigned to me.Thanks.

        Show
        venu Venu Uppalapati (Inactive) added a comment - Aruna, this is XDCR issue that was incorrectly assigned to me.Thanks.
        Hide
        apiravi Aruna Piravi added a comment -

        Thanks Venu, looks like Abhinav had closed it. Why was this reopened and by whom?

        Show
        apiravi Aruna Piravi added a comment - Thanks Venu, looks like Abhinav had closed it. Why was this reopened and by whom?
        Hide
        apiravi Aruna Piravi added a comment -

        Ok, looks like Abhinav left it as Fixed which Maria reassigned. Closing this issue, will reopen if seen in system tests.

        Show
        apiravi Aruna Piravi added a comment - Ok, looks like Abhinav left it as Fixed which Maria reassigned. Closing this issue, will reopen if seen in system tests.

          People

          • Assignee:
            apiravi Aruna Piravi
            Reporter:
            abhinav Abhinav Dangeti
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes