Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-5434

rebalance fails with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.2
    • Fix Version/s: 1.8.1
    • Component/s: ns_server
    • Security Level: Public
    • Labels:

      Description

      obvious from these logs :
      Rebalance exited with reason

      {wait_for_memcached_failed,"Authentication", ['ns_1@10.113.11.5','ns_1@10.113.11.11', 'ns_1@10.113.11.12','ns_1@10.113.11.16', 'ns_1@10.113.11.17','ns_1@10.113.11.19', 'ns_1@10.113.11.21','ns_1@10.113.11.23', 'ns_1@10.113.11.24','ns_1@10.113.11.27']}
      (repeated 1 times) ns_orchestrator002 ns_1@10.113.11.11 01:24:57 - Mon Jun 4, 2012
      Rebalance exited with reason {wait_for_memcached_failed,"Authentication", ['ns_1@10.113.11.5','ns_1@10.113.11.11', 'ns_1@10.113.11.12','ns_1@10.113.11.16', 'ns_1@10.113.11.17','ns_1@10.113.11.19', 'ns_1@10.113.11.21','ns_1@10.113.11.23', 'ns_1@10.113.11.24','ns_1@10.113.11.27']}

      ns_orchestrator002 ns_1@10.113.11.11 01:19:18 - Mon Jun 4, 2012
      Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
      'ns_1@10.113.11.3','ns_1@10.113.11.4',
      'ns_1@10.113.11.6','ns_1@10.113.11.7',
      'ns_1@10.113.11.8','ns_1@10.113.11.9',
      'ns_1@10.113.11.10','ns_1@10.113.11.13',
      'ns_1@10.113.11.14','ns_1@10.113.11.15',
      'ns_1@10.113.11.18','ns_1@10.113.11.20',
      'ns_1@10.113.11.22','ns_1@10.113.11.25',
      'ns_1@10.113.11.26','ns_1@10.113.11.28',
      'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
      'ns_1@10.113.11.11',
      'ns_1@10.113.11.12',
      'ns_1@10.113.11.16',
      'ns_1@10.113.11.17',
      'ns_1@10.113.11.19',
      'ns_1@10.113.11.21',
      'ns_1@10.113.11.23',
      'ns_1@10.113.11.24',
      'ns_1@10.113.11.27']
      ns_orchestrator004 ns_1@10.113.11.11 01:19:07 - Mon Jun 4, 2012
      Rebalance exited with reason stopped
      ns_orchestrator002 ns_1@10.113.11.11 01:13:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.17' for server shutdown ns_memcached002 ns_1@10.113.11.17 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.19' for server shutdown ns_memcached002 ns_1@10.113.11.19 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.23' for server shutdown ns_memcached002 ns_1@10.113.11.23 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.11' for server shutdown ns_memcached002 ns_1@10.113.11.11 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.12' for server shutdown ns_memcached002 ns_1@10.113.11.12 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.5' for server shutdown ns_memcached002 ns_1@10.113.11.5 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.16' for server shutdown ns_memcached002 ns_1@10.113.11.16 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.24' for server shutdown ns_memcached002 ns_1@10.113.11.24 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.21' for server shutdown ns_memcached002 ns_1@10.113.11.21 01:02:19 - Mon Jun 4, 2012
      Shutting down bucket "Authentication" on 'ns_1@10.113.11.27' for server shutdown ns_memcached002 ns_1@10.113.11.27 01:02:19 - Mon Jun 4, 2012
      Starting rebalance, KeepNodes = ['ns_1@10.113.11.1','ns_1@10.113.11.2',
      'ns_1@10.113.11.3','ns_1@10.113.11.4',
      'ns_1@10.113.11.6','ns_1@10.113.11.7',
      'ns_1@10.113.11.8','ns_1@10.113.11.9',
      'ns_1@10.113.11.10','ns_1@10.113.11.13',
      'ns_1@10.113.11.14','ns_1@10.113.11.15',
      'ns_1@10.113.11.18','ns_1@10.113.11.20',
      'ns_1@10.113.11.22','ns_1@10.113.11.25',
      'ns_1@10.113.11.26','ns_1@10.113.11.28',
      'ns_1@10.113.11.29','ns_1@10.113.11.30'], EjectNodes = ['ns_1@10.113.11.5',
      'ns_1@10.113.11.11',
      'ns_1@10.113.11.12',
      'ns_1@10.113.11.16',
      'ns_1@10.113.11.17',
      'ns_1@10.113.11.19',
      'ns_1@10.113.11.21',
      'ns_1@10.113.11.23',
      'ns_1@10.113.11.24',
      'ns_1@10.113.11.27']

      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        farshid Farshid Ghods (Inactive) created issue -
        farshid Farshid Ghods (Inactive) made changes -
        Field Original Value New Value
        Attachment Screen Shot 2012-06-04 at 1.30.06 AM.png [ 13478 ]
        farshid Farshid Ghods (Inactive) made changes -
        Summary rebalance waits with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt rebalance fails with wait_for_memcached if user stops rebalancing after one bucket was already rebalanced out because ns-server looks for "rebalanced out" buckets in the next rebalance attempt
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Truncated log event:
        {info_msg,<0.50.0>,
        {<0.29774.5803>,
        "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n",
        ['ns_1@10.113.11.11',<0.29774.5803>,ns_rebalancer,101,
        "Authentication",
        [

        {num_replicas,2},
        {ram_quota,104857600},
        {auth_type,sasl},
        {sasl_password,"soWeefeot7ai"},
        {type,membase},
        {num_vbuckets,1024},
        {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2', 'ns_1@10.113.11.3','ns_1@10.113.11.4', 'ns_1@10.113.11.6','ns_1@10.113.11.7', 'ns_1@10.113.11.8','ns_1@10.113.11.9', 'ns_1@10.113.11.10','ns_1@10.113.11.13', 'ns_1@10.113.11.14','ns_1@10.113.11.15', 'ns_1@10.113.11.18','ns_1@10.113.11.20', 'ns_1@10.113.11.22','ns_1@10.113.11.25', 'ns_1@10.113.11.26','ns_1@10.113.11.28', 'ns_1@10.113.11.29','ns_1@10.113.11.30']},
        {map,[['ns_1@10.113.11.10','ns_1@10.113.11.18', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.15', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.13', 'ns_1@10.113.11.28'], ['ns_1@10.113.11.15','ns_1@10.113.11.29', 'ns_1@10.113.11.9'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.3','ns_1@10.113.11.4','ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.7', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.8', 'ns_1@10.113.11.15'], ['ns_1@10.113.11.18','ns_1@10.113.11.29', 'ns_1@10.113.11.7'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.14','ns_1@10.113.11.18', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.4','ns_1@10.113.11.7','ns_1@10.113.11.9'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.9','ns_1@10.113.11.18'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.18','ns_1@10.113.11.1', 'ns_1@10.113.11.13'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.3','ns_1@10.113.11.7','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.1', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.15','ns_1@10.113.11.18', 'ns_1@10.113.11.2'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.7','ns_1@10.113.11.13', 'ns_1@10.113.11.30'], ['ns_1@10.113.11.13','ns_1@10.113.11.14', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.13','ns_1@10.113.11.15', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.1','ns_1@10.113.11.9','ns_1@10.113.11.13'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.22', 'ns_1@10.113.11.3'], ['ns_1@10.113.11.13','ns_1@10.113.11.25', 'ns_1@10.113.11.4'], ['ns_1@10.113.11.15','ns_1@10.113.11.28'|...], ['ns_1@10.113.11.15'|...], [...]|...]},
        {fastForwardMap,undefined}]]}}
        INFO REPORT <5881.164.0> 2012-06-04 04:19:07
        ===============================================================================

        config change:
        rebalance_status ->
        running

        INFO REPORT <5881.164.0> 2012-06-04 04:19:07
        ===============================================================================

        ns_node_disco_conf_events config all

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config done

        ERROR REPORT <5881.6.0> 2012-06-04 04:19:07
        ===============================================================================

        Truncated log event:
        {info_msg,<0.50.0>,
        {<0.164.0>,"config change:~n~p ->~n~s~n",
        [buckets,
        <<"[{configs,[{\"Authentication\",\n [{num_replicas,2}

        ,\n

        {ram_quota,104857600}

        ,\n

        {auth_type,sasl}

        ,\n

        {sasl_password,\"soWeefeot7ai\"}

        ,\n

        {type,membase}

        ,\n

        {num_vbuckets,1024}

        ,\n {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2',\n 'ns_1@10.113.11.3','ns_1@10.113.11.4',\n "...>>]}}
        INFO REPORT <5881.164.0> 2012-06-04 04:19:07
        ===============================================================================

        ns_node_disco_conf_events config all

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config done

        ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07
        ===============================================================================

        ns_1@10.113.11.11:<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication":
        {'EXIT',
        {noproc,
        {gen_server,call,
        [

        {'stats_reader-Authentication','ns_1@10.113.11.11'}

        ,

        {latest,minute}

        ]}}}

        INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08

        Show
        farshid Farshid Ghods (Inactive) added a comment - Truncated log event: {info_msg,<0.50.0>, {<0.29774.5803>, "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n", ['ns_1@10.113.11.11',<0.29774.5803>,ns_rebalancer,101, "Authentication", [ {num_replicas,2}, {ram_quota,104857600}, {auth_type,sasl}, {sasl_password,"soWeefeot7ai"}, {type,membase}, {num_vbuckets,1024}, {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2', 'ns_1@10.113.11.3','ns_1@10.113.11.4', 'ns_1@10.113.11.6','ns_1@10.113.11.7', 'ns_1@10.113.11.8','ns_1@10.113.11.9', 'ns_1@10.113.11.10','ns_1@10.113.11.13', 'ns_1@10.113.11.14','ns_1@10.113.11.15', 'ns_1@10.113.11.18','ns_1@10.113.11.20', 'ns_1@10.113.11.22','ns_1@10.113.11.25', 'ns_1@10.113.11.26','ns_1@10.113.11.28', 'ns_1@10.113.11.29','ns_1@10.113.11.30']}, {map,[['ns_1@10.113.11.10','ns_1@10.113.11.18', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.15', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.13', 'ns_1@10.113.11.28'], ['ns_1@10.113.11.15','ns_1@10.113.11.29', 'ns_1@10.113.11.9'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.3','ns_1@10.113.11.4','ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.7', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.8', 'ns_1@10.113.11.15'], ['ns_1@10.113.11.18','ns_1@10.113.11.29', 'ns_1@10.113.11.7'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.14','ns_1@10.113.11.18', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.4','ns_1@10.113.11.7','ns_1@10.113.11.9'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.9','ns_1@10.113.11.18'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.18','ns_1@10.113.11.1', 'ns_1@10.113.11.13'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.3','ns_1@10.113.11.7','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.1', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.15','ns_1@10.113.11.18', 'ns_1@10.113.11.2'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.7','ns_1@10.113.11.13', 'ns_1@10.113.11.30'], ['ns_1@10.113.11.13','ns_1@10.113.11.14', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.13','ns_1@10.113.11.15', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.1','ns_1@10.113.11.9','ns_1@10.113.11.13'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.22', 'ns_1@10.113.11.3'], ['ns_1@10.113.11.13','ns_1@10.113.11.25', 'ns_1@10.113.11.4'], ['ns_1@10.113.11.15','ns_1@10.113.11.28'|...], ['ns_1@10.113.11.15'|...], [...]|...]}, {fastForwardMap,undefined}]]}} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== config change: rebalance_status -> running INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.6.0> 2012-06-04 04:19:07 =============================================================================== Truncated log event: {info_msg,<0.50.0>, {<0.164.0>,"config change:~n~p ->~n~s~n", [buckets, <<"[{configs,[{\"Authentication\",\n [{num_replicas,2} ,\n {ram_quota,104857600} ,\n {auth_type,sasl} ,\n {sasl_password,\"soWeefeot7ai\"} ,\n {type,membase} ,\n {num_vbuckets,1024} ,\n {servers, ['ns_1@10.113.11.1','ns_1@10.113.11.2',\n 'ns_1@10.113.11.3','ns_1@10.113.11.4',\n "...>>] }} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07 =============================================================================== ns_1@10.113.11.11:<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication": {'EXIT', {noproc, {gen_server,call, [ {'stats_reader-Authentication','ns_1@10.113.11.11'} , {latest,minute} ]}}} INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        Truncated log event:
        {info_msg,<0.50.0>,
        {<0.29774.5803>,
        "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n",
        ['ns_1@10.113.11.11',<0.29774.5803>,ns_rebalancer,101,
        "Authentication",
        [

        {num_replicas,2},
        {ram_quota,104857600},
        {auth_type,sasl},
        {sasl_password,"soWeefeot7ai"},
        {type,membase},
        {num_vbuckets,1024},
        {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2', 'ns_1@10.113.11.3','ns_1@10.113.11.4', 'ns_1@10.113.11.6','ns_1@10.113.11.7', 'ns_1@10.113.11.8','ns_1@10.113.11.9', 'ns_1@10.113.11.10','ns_1@10.113.11.13', 'ns_1@10.113.11.14','ns_1@10.113.11.15', 'ns_1@10.113.11.18','ns_1@10.113.11.20', 'ns_1@10.113.11.22','ns_1@10.113.11.25', 'ns_1@10.113.11.26','ns_1@10.113.11.28', 'ns_1@10.113.11.29','ns_1@10.113.11.30']},
        {map,[['ns_1@10.113.11.10','ns_1@10.113.11.18', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.15', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.13', 'ns_1@10.113.11.28'], ['ns_1@10.113.11.15','ns_1@10.113.11.29', 'ns_1@10.113.11.9'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.3','ns_1@10.113.11.4','ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.7', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.8', 'ns_1@10.113.11.15'], ['ns_1@10.113.11.18','ns_1@10.113.11.29', 'ns_1@10.113.11.7'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.14','ns_1@10.113.11.18', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.4','ns_1@10.113.11.7','ns_1@10.113.11.9'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.9','ns_1@10.113.11.18'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.18','ns_1@10.113.11.1', 'ns_1@10.113.11.13'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.3','ns_1@10.113.11.7','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.1', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.15','ns_1@10.113.11.18', 'ns_1@10.113.11.2'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.7','ns_1@10.113.11.13', 'ns_1@10.113.11.30'], ['ns_1@10.113.11.13','ns_1@10.113.11.14', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.13','ns_1@10.113.11.15', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.1','ns_1@10.113.11.9','ns_1@10.113.11.13'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.22', 'ns_1@10.113.11.3'], ['ns_1@10.113.11.13','ns_1@10.113.11.25', 'ns_1@10.113.11.4'], ['ns_1@10.113.11.15','ns_1@10.113.11.28'|...], ['ns_1@10.113.11.15'|...], [...]|...]},
        {fastForwardMap,undefined}]]}}
        INFO REPORT <5881.164.0> 2012-06-04 04:19:07
        ===============================================================================

        config change:
        rebalance_status ->
        running

        INFO REPORT <5881.164.0> 2012-06-04 04:19:07
        ===============================================================================

        ns_node_disco_conf_events config all

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config done

        ERROR REPORT <5881.6.0> 2012-06-04 04:19:07
        ===============================================================================

        Truncated log event:
        {info_msg,<0.50.0>,
        {<0.164.0>,"config change:~n~p ->~n~s~n",
        [buckets,
        <<"[{configs,[{\"Authentication\",\n [{num_replicas,2}

        ,\n

        {ram_quota,104857600}

        ,\n

        {auth_type,sasl}

        ,\n

        {sasl_password,\"soWeefeot7ai\"}

        ,\n

        {type,membase}

        ,\n

        {num_vbuckets,1024}

        ,\n {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2',\n 'ns_1@10.113.11.3','ns_1@10.113.11.4',\n "...>>]}}
        INFO REPORT <5881.164.0> 2012-06-04 04:19:07
        ===============================================================================

        ns_node_disco_conf_events config all

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config

        INFO REPORT <5881.27798.1> 2012-06-04 04:19:07
        ===============================================================================

        Pushing config done

        ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07
        ===============================================================================

        ns_1@10.113.11.11:<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication":
        {'EXIT',
        {noproc,
        {gen_server,call,
        [

        {'stats_reader-Authentication','ns_1@10.113.11.11'}

        ,

        {latest,minute}

        ]}}}

        INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08

        Show
        farshid Farshid Ghods (Inactive) added a comment - Truncated log event: {info_msg,<0.50.0>, {<0.29774.5803>, "~s:~p:~s:~B: Rebalancing bucket ~p with config ~p~n", ['ns_1@10.113.11.11',<0.29774.5803>,ns_rebalancer,101, "Authentication", [ {num_replicas,2}, {ram_quota,104857600}, {auth_type,sasl}, {sasl_password,"soWeefeot7ai"}, {type,membase}, {num_vbuckets,1024}, {servers,['ns_1@10.113.11.1','ns_1@10.113.11.2', 'ns_1@10.113.11.3','ns_1@10.113.11.4', 'ns_1@10.113.11.6','ns_1@10.113.11.7', 'ns_1@10.113.11.8','ns_1@10.113.11.9', 'ns_1@10.113.11.10','ns_1@10.113.11.13', 'ns_1@10.113.11.14','ns_1@10.113.11.15', 'ns_1@10.113.11.18','ns_1@10.113.11.20', 'ns_1@10.113.11.22','ns_1@10.113.11.25', 'ns_1@10.113.11.26','ns_1@10.113.11.28', 'ns_1@10.113.11.29','ns_1@10.113.11.30']}, {map,[['ns_1@10.113.11.10','ns_1@10.113.11.18', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.10','ns_1@10.113.11.20', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.14', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.2','ns_1@10.113.11.15', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.10', 'ns_1@10.113.11.29'], ['ns_1@10.113.11.4','ns_1@10.113.11.13', 'ns_1@10.113.11.28'], ['ns_1@10.113.11.15','ns_1@10.113.11.29', 'ns_1@10.113.11.9'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.28','ns_1@10.113.11.3','ns_1@10.113.11.7'], ['ns_1@10.113.11.3','ns_1@10.113.11.4','ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.7', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.26','ns_1@10.113.11.8', 'ns_1@10.113.11.15'], ['ns_1@10.113.11.18','ns_1@10.113.11.29', 'ns_1@10.113.11.7'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.18','ns_1@10.113.11.3', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.9','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.6', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.29','ns_1@10.113.11.1','ns_1@10.113.11.3'], ['ns_1@10.113.11.14','ns_1@10.113.11.18', 'ns_1@10.113.11.25'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.1','ns_1@10.113.11.2','ns_1@10.113.11.10'], ['ns_1@10.113.11.4','ns_1@10.113.11.7','ns_1@10.113.11.9'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.9','ns_1@10.113.11.18'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.14','ns_1@10.113.11.1', 'ns_1@10.113.11.10'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.29','ns_1@10.113.11.9', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.18','ns_1@10.113.11.1', 'ns_1@10.113.11.13'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.4', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.3','ns_1@10.113.11.7','ns_1@10.113.11.14'], ['ns_1@10.113.11.22','ns_1@10.113.11.1', 'ns_1@10.113.11.14'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.28','ns_1@10.113.11.9', 'ns_1@10.113.11.22'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.4','ns_1@10.113.11.8','ns_1@10.113.11.13'], ['ns_1@10.113.11.2','ns_1@10.113.11.10', 'ns_1@10.113.11.18'], ['ns_1@10.113.11.15','ns_1@10.113.11.18', 'ns_1@10.113.11.2'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.6','ns_1@10.113.11.7','ns_1@10.113.11.8'], ['ns_1@10.113.11.7','ns_1@10.113.11.13', 'ns_1@10.113.11.30'], ['ns_1@10.113.11.13','ns_1@10.113.11.14', 'ns_1@10.113.11.1'], ['ns_1@10.113.11.13','ns_1@10.113.11.15', 'ns_1@10.113.11.20'], ['ns_1@10.113.11.1','ns_1@10.113.11.9','ns_1@10.113.11.13'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.20', 'ns_1@10.113.11.26'], ['ns_1@10.113.11.13','ns_1@10.113.11.22', 'ns_1@10.113.11.3'], ['ns_1@10.113.11.13','ns_1@10.113.11.25', 'ns_1@10.113.11.4'], ['ns_1@10.113.11.15','ns_1@10.113.11.28'|...], ['ns_1@10.113.11.15'|...], [...]|...]}, {fastForwardMap,undefined}]]}} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== config change: rebalance_status -> running INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.6.0> 2012-06-04 04:19:07 =============================================================================== Truncated log event: {info_msg,<0.50.0>, {<0.164.0>,"config change:~n~p ->~n~s~n", [buckets, <<"[{configs,[{\"Authentication\",\n [{num_replicas,2} ,\n {ram_quota,104857600} ,\n {auth_type,sasl} ,\n {sasl_password,\"soWeefeot7ai\"} ,\n {type,membase} ,\n {num_vbuckets,1024} ,\n {servers, ['ns_1@10.113.11.1','ns_1@10.113.11.2',\n 'ns_1@10.113.11.3','ns_1@10.113.11.4',\n "...>>] }} INFO REPORT <5881.164.0> 2012-06-04 04:19:07 =============================================================================== ns_node_disco_conf_events config all INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config INFO REPORT <5881.27798.1> 2012-06-04 04:19:07 =============================================================================== Pushing config done ERROR REPORT <5881.27838.1> 2012-06-04 04:19:07 =============================================================================== ns_1@10.113.11.11:<5881.27838.1>:ns_heart:156: Failed to get stats for bucket: "Authentication": {'EXIT', {noproc, {gen_server,call, [ {'stats_reader-Authentication','ns_1@10.113.11.11'} , {latest,minute} ]}}} INFO REPORT <5881.5131.7149> 2012-06-04 04:19:08
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Indeed we set set of bucket's list nodes to current list of nodes even if some of them are going to be rebalanced out. That's indeed going to start buckets on those previously nearly rebalanced out nodes. I think we can just reverse list of buckets before rebalancing as a workaround.

        This is the /diag/eval snipped I used to revers list of buckets:

        ns_config:update_key(buckets, fun ([

        {configs, Buckets0}

        ]) -> Buckets = lists:reverse(Buckets0), [

        {configs, Buckets}

        ] end).

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Indeed we set set of bucket's list nodes to current list of nodes even if some of them are going to be rebalanced out. That's indeed going to start buckets on those previously nearly rebalanced out nodes. I think we can just reverse list of buckets before rebalancing as a workaround. This is the /diag/eval snipped I used to revers list of buckets: ns_config:update_key(buckets, fun ([ {configs, Buckets0} ]) -> Buckets = lists:reverse(Buckets0), [ {configs, Buckets} ] end).
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        >>>ns_config:update_key(buckets, fun ([

        {configs, Buckets0}

        ]) -> Buckets = lists:reverse(Buckets0), [

        {configs, Buckets}

        ] end).

        what does bucket0 mean here ? should we pass the name of the bucket that is already rebalanced out ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - >>>ns_config:update_key(buckets, fun ([ {configs, Buckets0} ]) -> Buckets = lists:reverse(Buckets0), [ {configs, Buckets} ] end). what does bucket0 mean here ? should we pass the name of the bucket that is already rebalanced out ?
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        does it reverse by the alphabetical order or it reverses based on the state of the bucket ( is it rebalanced out or not yet )

        Show
        farshid Farshid Ghods (Inactive) added a comment - does it reverse by the alphabetical order or it reverses based on the state of the bucket ( is it rebalanced out or not yet )
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        it reverses current list. Buckets0 is current list.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - it reverses current list. Buckets0 is current list.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        I think we can do better than reversing. Give me few more minutes

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - I think we can do better than reversing. Give me few more minutes
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        This snipped rotates list of buckets instead:

        ns_config:update_key(buckets, fun ([

        {configs, Buckets}

        ]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)], [

        {configs, ShuffledBuckets}

        ] end).

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - This snipped rotates list of buckets instead: ns_config:update_key(buckets, fun ([ {configs, Buckets} ]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)] , [ {configs, ShuffledBuckets} ] end).
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        the problem with reversing is that if i have 5 buckets
        a,b,c,d,e,f
        first rebalance succeeds and remove "a"
        then rebalance is stuck and we restart rebalancing but fails with wait_for_memcached ("a")

        so reverse bucket list f,e,d,c,b,a

        rebalance moves f and then is stuck while moving e

        now reversing won't help anymore because both vbucket "f" and "a" are rebalanced out and the rest are still imbalanced

        Show
        farshid Farshid Ghods (Inactive) added a comment - the problem with reversing is that if i have 5 buckets a,b,c,d,e,f first rebalance succeeds and remove "a" then rebalance is stuck and we restart rebalancing but fails with wait_for_memcached ("a") so reverse bucket list f,e,d,c,b,a rebalance moves f and then is stuck while moving e now reversing won't help anymore because both vbucket "f" and "a" are rebalanced out and the rest are still imbalanced
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        exactly, So rotating would change a,b,c,d,e,f to b,c,d,e,f,a

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - exactly, So rotating would change a,b,c,d,e,f to b,c,d,e,f,a
        Hide
        farshid Farshid Ghods (Inactive) added a comment -

        >>ns_config:update_key(buckets, fun ([

        {configs, Buckets}

        ]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)], [

        {configs, ShuffledBuckets}

        ] end).

        so this script moves the buckets that are still part of the cluster to the head of list and the "rebalanced_out" ones to the end ?

        Show
        farshid Farshid Ghods (Inactive) added a comment - >>ns_config:update_key(buckets, fun ([ {configs, Buckets} ]) -> ShuffledBuckets = tl(Buckets) ++ [hd(Buckets)] , [ {configs, ShuffledBuckets} ] end). so this script moves the buckets that are still part of the cluster to the head of list and the "rebalanced_out" ones to the end ?
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        right.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - right.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Right. The idea is in worst case you can rebalance bucket after bucket. And then usual failover would just get rid of this nodes being in server list of buckets

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Right. The idea is in worst case you can rebalance bucket after bucket. And then usual failover would just get rid of this nodes being in server list of buckets
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -
        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - fixed for 1.8.1: http://review.couchbase.org/16783
        alkondratenko Aleksey Kondratenko (Inactive) made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #364 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/364/)
        MB-5434: don't add ejected nodes back to bucket's servers list (Revision 9ecbe6c6f1c596faf995dabc7e2abb3facd4c8b7)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/ns_rebalancer.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #364 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/364/ ) MB-5434 : don't add ejected nodes back to bucket's servers list (Revision 9ecbe6c6f1c596faf995dabc7e2abb3facd4c8b7) Result = SUCCESS Aliaksey Kandratsenka : Files : src/ns_rebalancer.erl
        Hide
        ketaki Ketaki Gangal added a comment -

        Tested on build 927 - Works fine. Seeing no error.

        Show
        ketaki Ketaki Gangal added a comment - Tested on build 927 - Works fine. Seeing no error.
        farshid Farshid Ghods (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes