Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-50355

[Windows] - Graceful failover + delta recovery + rebalance fails - {service_rebalance_failed,cbas, {agent_died,<26046.22190.1>, {linked_process_died,<26046.22191.1>, {no_connection,"cbas-service_api"}}}}.

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown
    • CX Sprint 278

    Description

      Steps to Repro
      1. Create a 4 node cluster with all services enabled.
      2. Import all the sample buckets.
      3. Enable TLS on 6.6.5 using following commands.

      1. curl -v -u Administrator:password -X POST http://localhost:8091/internalSettings -d "canEnableStrictEncryption=true"
      2. enable n2n encryption
      /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --enable
      3. Enforce it to strict
      curl -v -u Administrator:password http://localhost:8091/settings/security -d "clusterEncryptionLevel=strict"

      4. Disable TLS

      1. Bring it back to control from strict
      curl -v -u Administrator:password http://localhost:8091/settings/security -d "clusterEncryptionLevel=control"
      2. Disable n2n encryption
      /opt/couchbase/bin/couchbase-cli node-to-node-encryption -c http://localhost:8091 -u Administrator -p password --disable

      5. Do a graceful failover of one node(172.23.136.142).
      6. Do delta recovery of that node and rebalance.

      Rebalance failed as shown below.
      ns_1@172.23.136.137 11:45:25 PM 11 Jan, 2022

      Rebalance exited with reason {service_rebalance_failed,cbas,
      {agent_died,<26046.22190.1>,
      {linked_process_died,<26046.22191.1>,
      {no_connection,"cbas-service_api"}}}}.
      Rebalance Operation Id = 8bac2722070289d1b31b3c18f0aaedb3
      

      cbcollect_info attached. Retry of the failed rebalance worked fine.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.1.0-2065 contains cbas commit daf9248 with commit message:
            MB-50355: Avoid graceful shutdown on encryption level change

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2065 contains cbas commit daf9248 with commit message: MB-50355 : Avoid graceful shutdown on encryption level change

            Build couchbase-server-6.6.5-10076 contains cbas commit b960ba6 with commit message:
            MB-50355: Avoid graceful shutdown on encryption level change

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.5-10076 contains cbas commit b960ba6 with commit message: MB-50355 : Avoid graceful shutdown on encryption level change

            Validated this on 6.6.5 build 10078. Marking this closed.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Validated this on 6.6.5 build 10078. Marking this closed.

            Build couchbase-server-7.1.0-2078 contains cbas commit b960ba6 with commit message:
            MB-50355: Avoid graceful shutdown on encryption level change

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-2078 contains cbas commit b960ba6 with commit message: MB-50355 : Avoid graceful shutdown on encryption level change
            wayne Wayne Siu added a comment -

            Re-opening it for 7.0.4.

            wayne Wayne Siu added a comment - Re-opening it for 7.0.4.
            till Till Westmann added a comment - - edited

            Created a backport ticket for 7.0.4 (MB-51412), removed the 7.0.4 fixVersion from this ticket, and moved the approved-for-7.0.4 label to the new backport ticket.

            Re-resolving this ticket.

            till Till Westmann added a comment - - edited Created a backport ticket for 7.0.4 ( MB-51412 ), removed the 7.0.4 fixVersion from this ticket, and moved the approved-for-7.0.4 label to the new backport ticket. Re-resolving this ticket.

            Validated this on 7.1.0-2479. Marking this closed.

            Balakumaran.Gopal Balakumaran Gopal added a comment - Validated this on 7.1.0-2479. Marking this closed.

            Build couchbase-server-7.0.4-7220 contains cbas commit b960ba6 with commit message:
            MB-50355: Avoid graceful shutdown on encryption level change

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.4-7220 contains cbas commit b960ba6 with commit message: MB-50355 : Avoid graceful shutdown on encryption level change

            People

              Balakumaran.Gopal Balakumaran Gopal
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty