Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7286

UI logs showing multiple errors, (XDCR bi and uni, at the start of a rebalance-in operation): Server error during processing: ["web request failed" ...

    Details

      Description

      + 5 nodes rebalance in on each cluster
      Cluster setup: c1:c2::10:10
      biXDCR_bucket: c1 <---> c2
      uniXDCR_src: c1 ---> c2 :uniXDCR_dest
      Front end loads on c1 and c2 for biXDCR_bucket, and on c1 for uniXDCR_src.
      c1: http://ec2-177-71-230-72.sa-east-1.compute.amazonaws.com:8091/
      c2: http://ec2-175-41-186-167.ap-southeast-1.compute.amazonaws.com:8091/

      UI reports whole bunch of these errors at the start of rebalancing operation and with the front end loads:
      Server error during processing: ["web request failed",

      {path,"/pools/default"}

      ,

      {type,exit}

      ,
      {what,
      {timeout,

      {gen_server,call, [ns_doctor,get_tasks_version]}

      }},
      {trace,
      [

      {gen_server,call,2}

      ,

      {menelaus_web,build_pool_info,4}

      ,

      {menelaus_web,handle_pool_info_wait,5}

      ,

      {menelaus_web,loop,3}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]}] (repeated 1 times)

      Will attach grabbed diags from the particular server in a bit.

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        abhinav Abhinav Dangeti created issue -
        Hide
        abhinav Abhinav Dangeti added a comment -

        UI reports that this error seems to be on server node: ns_1@ec2-177-71-230-72.sa-east-1.compute.amazonaws.com
        Grabbed diags: https://s3.amazonaws.com/bugdb/MB-7286/ec2-177-71-230-72.sa-east-1.compute.amazonaws.com-8091-diag.txt.gz

        Show
        abhinav Abhinav Dangeti added a comment - UI reports that this error seems to be on server node: ns_1@ec2-177-71-230-72.sa-east-1.compute.amazonaws.com Grabbed diags: https://s3.amazonaws.com/bugdb/MB-7286/ec2-177-71-230-72.sa-east-1.compute.amazonaws.com-8091-diag.txt.gz
        Hide
        steve Steve Yen added a comment -

        from bug-scrub.

        ketaki: errors seem to go away after 10 minutes, but cluster is unaccessible?

        Show
        steve Steve Yen added a comment - from bug-scrub. ketaki: errors seem to go away after 10 minutes, but cluster is unaccessible?
        Hide
        steve Steve Yen added a comment -

        per-bug-scrub, moved to 2.0.1

        Show
        steve Steve Yen added a comment - per-bug-scrub, moved to 2.0.1
        steve Steve Yen made changes -
        Field Original Value New Value
        Labels 2.0-release-notes
        Fix Version/s 2.0.1 [ 10399 ]
        Fix Version/s 2.0 [ 10114 ]
        Priority Major [ 3 ] Critical [ 2 ]
        Hide
        kzeller kzeller added a comment -

        Added to RN:

        During a rebalance operation for clusters undergoing uni- and bi-directional replication
        via XDCR, the following server errors may appear, which are currently under
        investigation:

        Show
        kzeller kzeller added a comment - Added to RN: During a rebalance operation for clusters undergoing uni- and bi-directional replication via XDCR, the following server errors may appear, which are currently under investigation:
        Hide
        junyi Junyi Xie (Inactive) added a comment -

        Seems to me these errors usually mean the system is busy working on something which is pretty heavy (like rebalance, etc), and is unable to respond to UI request timely. Waiting for triage from ns_server team.

        Show
        junyi Junyi Xie (Inactive) added a comment - Seems to me these errors usually mean the system is busy working on something which is pretty heavy (like rebalance, etc), and is unable to respond to UI request timely. Waiting for triage from ns_server team.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )

        Show
        farshid Farshid Ghods (Inactive) added a comment - deferring to 2.1 per bug scrub meeting ( Dipti & Farshid -December 7th )
        farshid Farshid Ghods (Inactive) made changes -
        Fix Version/s 2.1 [ 10414 ]
        Fix Version/s 2.0.1 [ 10399 ]
        junyi Junyi Xie (Inactive) made changes -
        Component/s cross-datacenter-replication [ 10136 ]
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        This logs are insufficient.

        Please reproduce on 2.1.0

        5 <-> 5 XDCR is IMHO insane

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - This logs are insufficient. Please reproduce on 2.1.0 5 <-> 5 XDCR is IMHO insane
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Assignee Aliaksey Artamonau [ aliaksey artamonau ] Abhinav Dangeti [ abhinav ]
        abhinav Abhinav Dangeti made changes -
        Priority Critical [ 2 ] Major [ 3 ]
        Hide
        abhinav Abhinav Dangeti added a comment -

        Close for now, will reopen with additional logs if issue seen again.

        Show
        abhinav Abhinav Dangeti added a comment - Close for now, will reopen with additional logs if issue seen again.
        abhinav Abhinav Dangeti made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        maria Maria McDuff (Inactive) made changes -
        Assignee Abhinav Dangeti [ abhinav ] Venu Uppalapati [ venu ]
        venu Venu Uppalapati made changes -
        Assignee Venu Uppalapati [ venu ] Aruna Piravi [ apiravi ]
        Hide
        venu Venu Uppalapati added a comment -

        Aruna, this is XDCR issue that was incorrectly assigned to me.Thanks.

        Show
        venu Venu Uppalapati added a comment - Aruna, this is XDCR issue that was incorrectly assigned to me.Thanks.
        Hide
        apiravi Aruna Piravi added a comment -

        Thanks Venu, looks like Abhinav had closed it. Why was this reopened and by whom?

        Show
        apiravi Aruna Piravi added a comment - Thanks Venu, looks like Abhinav had closed it. Why was this reopened and by whom?
        Hide
        apiravi Aruna Piravi added a comment -

        Ok, looks like Abhinav left it as Fixed which Maria reassigned. Closing this issue, will reopen if seen in system tests.

        Show
        apiravi Aruna Piravi added a comment - Ok, looks like Abhinav left it as Fixed which Maria reassigned. Closing this issue, will reopen if seen in system tests.
        apiravi Aruna Piravi made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            apiravi Aruna Piravi
            Reporter:
            abhinav Abhinav Dangeti
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes