Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-9321

Implement new cluster orchestration (was: Get us off erlang's global facility and re-elect failed master quickly and safely

    XMLWordPrintable

Details

    • Sprint 2 - March 11 - April 3

    Description

      We have a number of bugs due to erlang global facility or related issue of not being able to spawn new master quickly. I.e.:

      • MB-7282 (erlang's global naming facility apparently drops globally registered service with actual service still alive (was: impossible to change settings/autoFailover after rebalance))
      • MB-7168 [Doc'd 2.2.0] failover of node that's completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node)
      • MB-8682 start rebalance request is hunging sometimes (looks like another global facility issue)
      • MB-5622 Crash of master node may lead to autofailover in 2 minutes instead of configured shorter autofailover period or similarly slow manual failover

      By getting us off global, we will fix all this issues.

      Attachments

        Issue Links

          blocks

          Improvement - An improvement or enhancement to an existing feature or task. MB-12739 Improve Auto-failover for RZA

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-7282 erlang's global naming facility apparently drops globally registered service with actual service still alive (was: impossible to change settings/autoFailover after rebalance)

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-9691 rebalance repeated failed when add nodes back into cluster

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-5622 Crash of master node may lead to autofailover in 2 minutes instead of configured shorter autofailover period or similarly slow manual failover

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. MB-9415 auto-failover in seconds - (reduced from minimum 30 seconds)

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-9691 rebalance repeated failed when add nodes back into cluster

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-14967 Spend some time looking for a workaround for a problem with erlang global facility (was: [system test] Rebalance In fails with error " Request failed/" errors.)

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-22807 Failover of node taking ~15 sec when the node down is orchestrator node and timeout is 5 sec

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MB-11614 Discussion - Should we move auto-failover out of erlang?

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MB-9066 Increase Autofailover Counter: Enable setting the number of auto-failovers allowed on cluster

          • Major - Major loss of function.
          • Resolved
          For Gerrit Dashboard: MB-9321
          # Subject Branch Project Status CR V
          86335,11 MB-9321 New global name registry. master ns_server Status: MERGED +2 +1
          88124,17 MB-9321 Infrastructure for new orchestration. master ns_server Status: MERGED +2 +1
          88644,19 MB-9321 Use leader_activities in ns_janitor. master ns_server Status: MERGED -1 +1
          88645,19 MB-9321 Use leader_activities in service_janitor. master ns_server Status: MERGED -1 +1
          88646,19 MB-9321 Use leader_activities for failover. master ns_server Status: MERGED -1 +1
          88647,19 MB-9321 Use leader_activities for rebalance. master ns_server Status: MERGED +2 +1
          88648,19 MB-9321 Use leader_activities for graceful failover. master ns_server Status: MERGED +2 +1
          88660,22 MB-9321 Register recovery_server as leader activity. master ns_server Status: MERGED +2 +1

          Activity

            People

              Aliaksey Artamonau Aliaksey Artamonau (Inactive)
              alkondratenko Aleksey Kondratenko (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  PagerDuty