Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50773

Analytics is not accessible from the upgraded node in mixed mode cluster

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Not a Bug
    • 7.1.0
    • 7.1.0
    • analytics
    • Enterprise Edition 7.1.0 build 2215

    Description

      Steps to reproduce -

      1. Have a 4 node cluster as follows -
      Node Services         CPU_utilization Mem_total Mem_free   Swap_mem_used         Active / Replica   Version              
      10.112.205.102 index, kv, n1ql 54.1622760801   1.80 GiB   1.16 GiB   0.0 Byte / 1.03 GiB   5000 / 5000       7.0.3-7031-enterprise
      10.112.205.104 cbas             55.1369863014   1.80 GiB   919.89 MiB 0.0 Byte / 1.03 GiB   0 / 0             7.0.3-7031-enterprise
      10.112.205.103 cbas             1.73116089613   1.80 GiB   784.16 MiB 126.06 MiB / 1.03 GiB 0 / 0             7.0.3-7031-enterprise
      10.112.205.101 index, kv, n1ql 70.3974895397   1.80 GiB   1.19 GiB   121.04 MiB / 1.03 GiB 5000 / 5000       7.0.3-7031-enterprise

      2. create a bucket and load some data into it.

      3. Create dataverses, datasets and indexes in cbas.

      4. let the ingestion into datasets complete.

      5. Gracefully failover one of the kv node.

      6. Upgrade the failed over node to 7.1.0-2215 build.

      7. Set recovery type as delta for failed over node and add it back and rebalance. Cluster would look like below

      Node Services         CPU_utilization Mem_total Mem_free   Swap_mem_used         Active / Replica   Version              
      10.112.205.102 index, kv, n1ql 54.1622760801   1.80 GiB   1.16 GiB   0.0 Byte / 1.03 GiB   5000 / 5000       7.0.3-7031-enterprise
      10.112.205.104 cbas             55.1369863014   1.80 GiB   919.89 MiB 0.0 Byte / 1.03 GiB   0 / 0             7.0.3-7031-enterprise
      10.112.205.103 cbas             1.73116089613   1.80 GiB   784.16 MiB 126.06 MiB / 1.03 GiB 0 / 0             7.0.3-7031-enterprise
      10.112.205.101 index, kv, n1ql 70.3974895397   1.80 GiB   1.19 GiB   121.04 MiB / 1.03 GiB 5000 / 5000       7.1.0-2215-enterprise

      8. Once the rebalance completes, try accessing analytics.

      Following error is seen -

      Received error 503 when contacting the analytics service. Try refreshing the browser, as errors may occur if the analytics service is still warming up or suffering a transient error.

      9. Follow steps 5-8 again for the other KV node. Same error is seen for analytics.

      Node Services         CPU_utilization Mem_total Mem_free   Swap_mem_used         Active / Replica   Version              
      10.112.205.102 index, kv, n1ql 54.1622760801   1.80 GiB   1.16 GiB   0.0 Byte / 1.03 GiB   5000 / 5000       7.1.0-2215-enterprise
      10.112.205.104 cbas             55.1369863014   1.80 GiB   919.89 MiB 0.0 Byte / 1.03 GiB   0 / 0             7.0.3-7031-enterprise
      10.112.205.103 cbas             1.73116089613   1.80 GiB   784.16 MiB 126.06 MiB / 1.03 GiB 0 / 0             7.0.3-7031-enterprise
      10.112.205.101 index, kv, n1ql 70.3974895397   1.80 GiB   1.19 GiB   121.04 MiB / 1.03 GiB 5000 / 5000       7.1.0-2215-enterprise

       

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          Umang,

          I believe the issue is related to ns_server not proxying requests to nodes with different version than the node from which you are accessing the UI. It worked when I upgraded one of the CBAS nodes to 7.1.0.

          Assigning to ns_server team to confirm that this is the expected behaviour.

          ali.alsuliman Ali Alsuliman added a comment - Umang , I believe the issue is related to ns_server not proxying requests to nodes with different version than the node from which you are accessing the UI. It worked when I upgraded one of the CBAS nodes to 7.1.0. Assigning to ns_server team to confirm that this is the expected behaviour.
          dfinlay Dave Finlay added a comment -

          Hi Ali:

          Thanks, yes, exactly right. We only proxy requests to nodes with the same version (as generally we can't assume that the pluggable UI and REST APIs perfectly honor the cluster compat version).

          dfinlay Dave Finlay added a comment - Hi Ali: Thanks, yes, exactly right. We only proxy requests to nodes with the same version (as generally we can't assume that the pluggable UI and REST APIs perfectly honor the cluster compat version).
          umang.agrawal Umang added a comment -

          Verified that CBAS is accessible from UI when one of the CBAS nodes in upgraded. Verified on build 7.1.0-2250

          umang.agrawal Umang added a comment - Verified that CBAS is accessible from UI when one of the CBAS nodes in upgraded. Verified on build 7.1.0-2250

          People

            umang.agrawal Umang
            umang.agrawal Umang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty