Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-56060

[System test upgrade] : Online upgrade from 7.1.4-3601 -> 7.2.0-5263 using swap rebalance continuously fails with analytics service

    XMLWordPrintable

Details

    • Untriaged
    • Centos 64-bit
    • 0
    • No
    • Analytics Sprint 16

    Description

      Steps to Repro
      1. Run neo longevity test for 3 days on 7.1.4-3601.

      ./sequoia -client 172.23.104.254:2375 -provider file:centos_third_cluster.yml -test tests/integration/neo/test_neo.yml -scope tests/integration/neo/scope_neo_magma.yml -scale 3 -repeat 0 -log_level 0 -version 7.1.4-3601 -skip_setup=false -skip_test=false -skip_teardown=true -skip_cleanup=false -continue=false -collect_on_error=false -stop_on_error=false -duration=604800 -show_topology=true
      

      2. Try upgrading to 7.2.0-5263 using rebalance in/out and swap rebalance strategies.

      Rebalance button was still enabled post rebalance. On analysing the logs we saw that the analytics rebalance did not complete successfully. Tried couple of times and got the same error.

      172.23.108.138 10:24:57 AM 20 Mar, 2023

      Analytics Service unable to successfully rebalance eb0007f5063099cc9ea4c32da9f566d0 due to 'CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1743678. User action: Try again later'; see analytics_info.log for details
      

      172.23.108.138 7:26:07 PM 20 Mar, 2023

      Analytics Service unable to successfully rebalance 64c8e0d2547f215501acc0266eb08e36 due to 'CBAS0001: Analytics collections in different partitions have different DCP states. Mutations needed to catch up = 1737994. User action: Try again later'; see analytics_info.log for details
      

      cbcollect_info attached.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty