Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-51840

[Upgrade] Local Links are disconnected post upgrading from 6.5.2 to 7.1.0

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Won't Fix
    • 7.1.0
    • None
    • analytics
    • 7.1.0-2549
    • Untriaged
    • Centos 64-bit
    • 1
    • No

    Description

      Scenario -
      Offline upgrade from 6.5.2 to 7.1.0

      Test Steps -
      1. Create cluster as follows -

      Node Services CPU_utilization Mem_total Mem_free Swap_mem_used Active / Replica Version
      172.23.121.243 cbas 0.501253132832 3.91 GiB 3.06 GiB 0.0 Byte / 3.50 GiB 0 / 0 6.5.2-6634-enterprise
      172.23.105.31 index, kv, n1ql 0.501253132832 3.91 GiB 3.39 GiB 66.75 MiB / 3.50 GiB 0 / 0 6.5.2-6634-enterprise
      172.23.105.244 index, kv, n1ql 1.25944584383 3.91 GiB 3.41 GiB 39.00 MiB / 3.50 GiB 0 / 0 6.5.2-6634-enterprise
      172.23.105.245 cbas 2.01511335013 3.91 GiB 3.13 GiB 78.25 MiB / 3.50 GiB 0 / 0 6.5.2-6634-enterprise
      172.23.121.244 cbas 0.503778337531 3.91 GiB 2.99 GiB 0.0 Byte / 3.50 GiB 0 / 0 6.5.2-6634-enterprise

      2. Create a bucket and load 10K docs.
      3. Create dataverses and datasets on cbas.
      4. Wait for ingestion to finish.
      5. Failover a node, upgrade it using rpm -u and then add it back to cluster.
      6. Do this for all the nodes of cluster.
      7. Once all the nodes are upgraded, rebalance once again to activate analytics.
      8. The Local link for one of the dataverse apart from Default is disconnected.

      Other observations -
      I tried 2 scenarios for the above test.
      1. 2 Dataverses (including Deafult) and 4 datasets in each dataverse.
      2. 4 Dataverses (including Deafult) and 2 datasets in each dataverse.
      Above infra is created Pre-upgrade.

      Now in 1st case, the Local link is always disconnected for the non-Default dataverse.
      For 2nd case, all the Local links are connected.

      Note- No error is thrown if we explicitly connect the disconnected link and continue the test further.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          umang.agrawal Umang added a comment - Upgrade from 6.6.4-9961 to 7.1.0-2549 https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T045335-ns_1%40172.23.105.244.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T045335-ns_1%40172.23.105.245.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T045335-ns_1%40172.23.105.31.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T045335-ns_1%40172.23.121.243.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T045335-ns_1%40172.23.121.244.zip
          umang.agrawal Umang added a comment - Upgrade from 6.5.2-6634 to 7.1.0-2549 https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T074947-ns_1%40172.23.105.244.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T074947-ns_1%40172.23.105.245.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T074947-ns_1%40172.23.105.31.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T074947-ns_1%40172.23.121.243.zip https://cb-jira.s3.us-east-2.amazonaws.com/logs/MB-51840/collectinfo-2022-04-18T074947-ns_1%40172.23.121.244.zip

          When did the test pass last time?

          till Till Westmann added a comment - When did the test pass last time?
          umang.agrawal Umang added a comment -

          This test passed on build 7.1.0-2534.

          umang.agrawal Umang added a comment - This test passed on build 7.1.0-2534.
          michael.blow Michael Blow added a comment -

          >>8. The Local link for one of the dataverse apart from Default is disconnected.

          Is this an automated test?

          How did the test framework (or you) determine that the link is disconnected?

          >> I tried 2 scenarios for the above test.
          >> 1. 2 Dataverses (including Deafult) and 4 datasets in each dataverse.
          >> 2. 4 Dataverses (including Deafult) and 2 datasets in each dataverse.

          To which scenario does the set of 6.5.2 logs attached correspond?

          michael.blow Michael Blow added a comment - >>8. The Local link for one of the dataverse apart from Default is disconnected. Is this an automated test? How did the test framework (or you) determine that the link is disconnected? >> I tried 2 scenarios for the above test. >> 1. 2 Dataverses (including Deafult) and 4 datasets in each dataverse. >> 2. 4 Dataverses (including Deafult) and 2 datasets in each dataverse. To which scenario does the set of 6.5.2 logs attached correspond?
          michael.blow Michael Blow added a comment -

          This is due to how analytics managed link state prior to 6.6.x. The bucket in the other dataverse was disconnected due to the failover of the KV node as part of the upgrade sequence. As a result, after the upgrade of the analytics nodes, the link associated with this disconnected bucket was not connected.

          Ås the state is becoming disconnected in 6.5.2, there would be no way to address this in 7.1.1. Note, this is not a regression. I would suggest we release note this as an upgrade note when upgrading from prior to 6.6.x.

          michael.blow Michael Blow added a comment - This is due to how analytics managed link state prior to 6.6.x. The bucket in the other dataverse was disconnected due to the failover of the KV node as part of the upgrade sequence. As a result, after the upgrade of the analytics nodes, the link associated with this disconnected bucket was not connected. Ås the state is becoming disconnected in 6.5.2, there would be no way to address this in 7.1.1. Note, this is not a regression. I would suggest we release note this as an upgrade note when upgrading from prior to 6.6.x.
          till Till Westmann added a comment - - edited

          Summary: Known Issue

          When upgrading from a version earlier than 6.6.0 to 6.6.0 or later using failover the Local link(s) to the Analytics Service may be disconnected.

          Workaround:

          Reconnect the Local link(s) manually.

          till Till Westmann added a comment - - edited Summary: Known Issue When upgrading from a version earlier than 6.6.0 to 6.6.0 or later using failover the Local link(s) to the Analytics Service may be disconnected. Workaround : Reconnect the Local link(s) manually.

          People

            umang.agrawal Umang
            umang.agrawal Umang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty