Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-24820

Couchbase server is not usable after autofailover of node due to network failure.

    XMLWordPrintable

Details

    Description

      The couchbase server is unusable immediately after the autofailover of a node due to network failure. The issue is seen when the timeout period is > 30 sec (120 sec in the test). The node(CB) is unreachable immediately after the autofailover and remains unreachable till the couchbase server is restarted on the node.

      Run the following test to reproduce the issue.
      Create an ini file with atleast 4 nodes (need vms or physical machines since test use commands to stop network of the node). Example for an ini file can be found under b/resources in the testrunner repo.
      Run the following test from test runner.
      ./testrunner -i <ini file> -t failover.AutoFailoverTests.AutoFailoverTests.test_autofailover,timeout=120,num_node_failures=1,failover_action=restart_network,nodes_init=3,num_items=10000

      The test might pass but the node(CB server) that is failed over (the second node in the ini file) is not reachable (ping to http://<ip>:8091/nodes/self fails causing all subsequent tests to fail). Restarting the server on the machine is the only option to reach the couchbase server.

      This issue is causing major failures on network failure tests for autofailover job. http://qa.sc.couchbase.com/view/nserver/job/cen006-nserv-autofailover-networkfailure/37/consoleFull is one example for the same. More jobs can be found at http://qa.sc.couchbase.com/view/nserver/job/cen006-nserv-autofailover-networkfailure/ if above link is unaccessible(jobs are deleted periodically)

      This is a regression seen since 5.0.0-2008 weekly build (the tests had passed in the prior weeks builds)

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            bharath.gp Bharath G P
            bharath.gp Bharath G P
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty