Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5298

Rebalance failed with reason {case_clause,{{ok,replica},{ok,replica}}} when rebalancing out a node which was failed over due to netwrok connectivity issues but it re-appears while rebalancing

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.1-release-candidate
    • Fix Version/s: 2.0
    • Component/s: ns_server
    • Security Level: Public
    • Environment:
      1.8.1-815-rel

      Description

      Failing testcase
      failovertests.FailoverTests.test_failover_firewall,replica=3,load_ratio=10

      [ns_server:info] [2012-05-14 13:26:46] [ns_1@10.1.3.55:<0.3846.2>:ns_janitor:wait_for_memcached:286] Waiting for "default" on ['ns_1@10.1.3.50','ns_1@10.1.3.51','ns_1@10.1.3.52',
      'ns_1@10.1.3.54']
      [ns_server:debug] [2012-05-14 13:26:46] [ns_1@10.1.3.55:ns_bucket_worker:ns_bucket_sup:update_childs:91] Stopping child for dead bucket: {{per_bucket_sup,"default"},
      <0.23773.0>,supervisor,
      [single_bucket_sup]}

      [ns_server:debug] [2012-05-14 13:26:46] [ns_1@10.1.3.55:<0.23773.0>:single_bucket_sup:top_loop:28] Delegating exit

      {'EXIT',<0.23699.0>,shutdown}

      to child supervisor: <0.23774.0>

      [error_logger:error] [2012-05-14 13:26:03] [ns_1@10.1.3.50:error_logger:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_vbucket_mover:init/1
      pid: <0.29733.2>
      registered_name: []
      exception exit: {case_clause,{{ok,replica},

      {ok,replica}}}
      in function gen_server:terminate/6
      ancestors: [<0.28176.2>]
      messages: [{'EXIT',<0.19829.3>,
      {exited,
      {'EXIT',<0.29733.2>,
      {case_clause,{{ok,replica}

      ,{ok,replica}}}}}},
      {'EXIT',<0.19792.3>,
      {exited,
      {'EXIT',<0.29733.2>,
      {case_clause,ok,replica},{ok,replica}}}},
      {'EXIT',<0.15598.3>,
      {exited,
      {'EXIT',<0.29733.2>,
      {case_clause,ok,replica},{ok,replica}}}}]

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        your specific way of using firewall makes this problem probable to observe. That's my understanding.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - your specific way of using firewall makes this problem probable to observe. That's my understanding.
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        firewall is our way of simulating a node disappearing and re-appearing. we can simulate that by also shutting down the network interface or pulling the network cable if it helps

        Show
        farshid Farshid Ghods (Inactive) added a comment - firewall is our way of simulating a node disappearing and re-appearing. we can simulate that by also shutting down the network interface or pulling the network cable if it helps
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        I have evidence that you're enabling firewall back in some very specific way. Particularly memcached traffic is re-enabled first. And then minutes later you re-enable erlang traffic.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - I have evidence that you're enabling firewall back in some very specific way. Particularly memcached traffic is re-enabled first. And then minutes later you re-enable erlang traffic.
        Hide
        dipti Dipti Borkar added a comment -

        will defer to later release.

        Show
        dipti Dipti Borkar added a comment - will defer to later release.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Cannot happen on replicator on destination.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Cannot happen on replicator on destination.

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes