Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5279

user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.8.1-release-candidate
    • Fix Version/s: 1.8.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
    • Environment:
      CentOS 64 bit

      1.8.1-807

      Description

      Have not seen this issue previously ever on 181. Seems to be something new.

      Test case that failed
      swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=3,num-buckets=5,num-swap=2,swap-orchestrator=True,keys-count=800000

      1) Failover 2 nodes, including the orchestrator
      2) Rebalance them out
      3) Add them back and rebalance in, without cleaning them.

      [rebalance:error] [2012-05-10 8:48:59] [ns_1@10.1.3.74:<0.27930.16>:ns_rebalancer:do_wait_buckets_shutdown:170] Failed to wait deletion of some buckets on some nodes: [{'ns_1@10.1.3.77',
      {'EXIT',

      {old_buckets_shutdown_wait_failed, ["bucket-2", "bucket-1", "bucket-0", "bucket-4", "bucket-3"]}

      }}]

      [user:info] [2012-05-10 8:48:59] [ns_1@10.1.3.74:<0.27893.16>:ns_orchestrator:handle_info:246] Rebalance exited with reason {buckets_shutdown_wait_failed,
      [{'ns_1@10.1.3.77',
      {'EXIT',

      {old_buckets_shutdown_wait_failed, ["bucket-2","bucket-1","bucket-0", "bucket-4","bucket-3"]}

      }}]}

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        ok so if this is a force kill then we dont have to fix this on ep-engine.

        for this specific issue all we care about it the end user experience and if user will be able to add the node back to the cluster without any issues and your patch takes care of it so we can close this bug

        Show
        farshid Farshid Ghods (Inactive) added a comment - ok so if this is a force kill then we dont have to fix this on ep-engine. for this specific issue all we care about it the end user experience and if user will be able to add the node back to the cluster without any issues and your patch takes care of it so we can close this bug
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Patches backported and uploaded to gerrit. I can rubberstamp ns_server's patches. But not memcached.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Patches backported and uploaded to gerrit. I can rubberstamp ns_server's patches. But not memcached.
        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - http://review.couchbase.org/16432 http://review.couchbase.org/16436 http://review.couchbase.org/16437 http://review.couchbase.org/16438
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Should be fixed now with memcached & ns_server side merged

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Should be fixed now with memcached & ns_server side merged
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #357 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/357/)
        MB-5279:bp: allow port_servers to send arbitrary data to port (Revision ab2b8ecf77658fda7266a9eaed1b474f29741e43)
        MB-5279:bp:made supervisor_cushion exit with same reason as child (Revision 5ba3d6e1634c613ff3b6c1678f3eb04449434834)
        MB-5279:bp: ask memcached to die real quick on failover (Revision 2f14d3a4107b6c7fccc4c479832048bb259f469c)
        MB-5279: send shutdown command instead of empty line to memcached (Revision 6118863758b88da374868d2adf7a890096067394)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/ns_port_server.erl

        Aliaksey Kandratsenka :
        Files :

        • src/supervisor_cushion.erl

        Aliaksey Kandratsenka :
        Files :

        • src/ns_port_sup.erl
        • src/ns_server_sup.erl

        Aliaksey Kandratsenka :
        Files :

        • src/ns_port_server.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #357 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/357/ ) MB-5279 :bp: allow port_servers to send arbitrary data to port (Revision ab2b8ecf77658fda7266a9eaed1b474f29741e43) MB-5279 :bp:made supervisor_cushion exit with same reason as child (Revision 5ba3d6e1634c613ff3b6c1678f3eb04449434834) MB-5279 :bp: ask memcached to die real quick on failover (Revision 2f14d3a4107b6c7fccc4c479832048bb259f469c) MB-5279 : send shutdown command instead of empty line to memcached (Revision 6118863758b88da374868d2adf7a890096067394) Result = SUCCESS Aliaksey Kandratsenka : Files : src/ns_port_server.erl Aliaksey Kandratsenka : Files : src/supervisor_cushion.erl Aliaksey Kandratsenka : Files : src/ns_port_sup.erl src/ns_server_sup.erl Aliaksey Kandratsenka : Files : src/ns_port_server.erl

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            karan Karan Kumar (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: