Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48411

Waiting for non-tls web servers restart is broken in node addition (caused by ns_config_event -> chronicle_compat_events switch in menelaus_event)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 7.0.0, Neo, 7.0.2, 7.0.1
    • 7.0.2
    • ns_server
    • Untriaged
    • 1
    • Unknown

    Description

      menelaus_event used to subscribe to ns_config_events, but it was changed to chronicle_compat_events. After that change menelaus_event:sync(ns_config_event) stopped working, which triggers the following race condition:
      in node addition scenario, it might happen that web server is still restarting, when the node-being-added receives completeJoin http req.

      UPDATE: node-init scenario is also affected (when ipfamily is being flipped during init)

      Attachments

        For Gerrit Dashboard: MB-48411
        # Subject Branch Project Status CR V

        Activity

          Build couchbase-server-7.0.2-6668 contains ns_server commit f09935e with commit message:
          MB-48411: Sync with proper menelaus_event handler

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6668 contains ns_server commit f09935e with commit message: MB-48411 : Sync with proper menelaus_event handler
          ritam.sharma Ritam Sharma added a comment -

          Timofey Barmin - Please help with steps to validate the issue.

          ritam.sharma Ritam Sharma added a comment - Timofey Barmin - Please help with steps to validate the issue.
          timofey.barmin Timofey Barmin added a comment - - edited

          It's a race condition, so it's hard to reproduce the issue. I actually tested the race condition itself, you need to confirm that node addition via http works (basically that I didn't break it).
          In order to do that you need:
          1) provision nodes with certificates (needed because we don't allow node addition via http with self-generated certs)
          2) specify http://<nodeName> in the address field when adding the node

          timofey.barmin Timofey Barmin added a comment - - edited It's a race condition, so it's hard to reproduce the issue. I actually tested the race condition itself, you need to confirm that node addition via http works (basically that I didn't break it). In order to do that you need: 1) provision nodes with certificates (needed because we don't allow node addition via http with self-generated certs) 2) specify http://<nodeName> in the address field when adding the node

          Build couchbase-server-7.1.0-1284 contains ns_server commit f09935e with commit message:
          MB-48411: Sync with proper menelaus_event handler

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1284 contains ns_server commit f09935e with commit message: MB-48411 : Sync with proper menelaus_event handler
          ritam.sharma Ritam Sharma added a comment -

          Timofey Barmin - Thank you for the steps. Closing out this issue with build - 7.0.2-6670.
          1) provision nodes with certificates (needed because we don't allow node addition via http with self-generated certs)
          2) specify http://<nodeName> in the address field when adding the node
          Above scenarios are pass the validation.

          ritam.sharma Ritam Sharma added a comment - Timofey Barmin - Thank you for the steps. Closing out this issue with build - 7.0.2-6670. 1) provision nodes with certificates (needed because we don't allow node addition via http with self-generated certs) 2) specify http://<nodeName> in the address field when adding the node Above scenarios are pass the validation.

          People

            sumedh.basarkod Sumedh Basarkod
            timofey.barmin Timofey Barmin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty