Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6165

Enabling use of index pausing during rebalance crashes set view group

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0-beta
    • Component/s: ns_server, view-engine
    • Security Level: Public
    • Labels:
      None

      Description

      Running with my consistent views patches and enabled (initially disabled) index pausing:

      1. rlwrap ../send-diag-eval lh:9000 -q
        ns_config:set(index_pausing_disabled, false).
        ok
        #

      The following rebalance failed:

      Rebalance exited with reason {{{{function_clause,
      [{orddict,fetch,
      [53,
      [

      {96,9852}

      ,

      {97,16395}

      ,

      {98,16420}

      ,

      {99,9837}

      ,

      {100,9837}

      ,

      {101,22988}

      ,

      {102,16395}

      ,

      {103,9852}

      ,

      {104,9855}

      ,

      {105,16385}

      ]]},

      {couch_set_view_group, '-process_mark_as_unindexable/2-fun-0-',3}

      ,

      {lists,foldl,3}

      ,

      {couch_set_view_group, process_mark_as_unindexable,2}

      ,

      {couch_set_view_group,handle_call,3}

      ,

      {gen_server,handle_msg,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]},
      {gen_server,call,
      [<0.614.0>,

      {mark_as_unindexable,"5"}

      ,
      infinity]}},
      {gen_server,call,
      ['capi_set_view_manager-default',

      {set_vbucket_states, [active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,active,active,active,active, active,active,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,active,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,missing,missing,missing,missing, missing,active,active,active,active,active, active,active,active,active,active,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, replica,replica,replica,replica,replica, missing], [undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,paused,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined, undefined,undefined,undefined,undefined]}

      ,
      infinity]}},
      {gen_server,call,
      [

      {'janitor_agent-default','n_0@10.17.2.163'}

      ,
      {if_rebalance,<0.2974.0>,
      {update_vbucket_state,53,active,paused,
      undefined}},
      60000]}}

      1. diag.xz
        810 kB
        Aleksey Kondratenko
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        You asked for the same partition to be unindexable twice.
        The second time it crashes because it's already marked as unindexable.

        Added a change to gerrit to ignore such case: http://review.couchbase.org/#change,19403

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - You asked for the same partition to be unindexable twice. The second time it crashes because it's already marked as unindexable. Added a change to gerrit to ignore such case: http://review.couchbase.org/#change,19403
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-couchdb-preview #473 (See http://qa.hq.northscale.net/job/github-couchdb-preview/473/)
        MB-6165 Ignore index pausing of already paused partitions (Revision fc3a87e487f641ea80e56c4a48fd284442ee7388)

        Result = SUCCESS
        Filipe David Borba Manana :
        Files :

        • src/couch_set_view/src/couch_set_view_group.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-couchdb-preview #473 (See http://qa.hq.northscale.net/job/github-couchdb-preview/473/ ) MB-6165 Ignore index pausing of already paused partitions (Revision fc3a87e487f641ea80e56c4a48fd284442ee7388) Result = SUCCESS Filipe David Borba Manana : Files : src/couch_set_view/src/couch_set_view_group.erl
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Thanks Filipe. I think there's similar crash now:

        Event Module Code Server Node Time
        Shutting down bucket "default" on 'n_0@10.17.2.163' for server shutdown (repeated 1 times) ns_memcached002 n_0@10.17.2.163 14:29:18 - Thu Aug 9, 2012
        Bucket "default" loaded on node 'n_0@10.17.2.163' in 0 seconds. ns_memcached001 n_0@10.17.2.163 14:29:03 - Thu Aug 9, 2012
        Bucket "default" loaded on node 'n_0@10.17.2.163' in 1 seconds. ns_memcached001 n_0@10.17.2.163 14:29:01 - Thu Aug 9, 2012
        Rebalance exited with reason {{bulk_set_vbucket_state_failed,
        [{'n_0@10.17.2.163',
        {'EXIT',
        {{{bad_return_value,
        {error,
        <<"Intersection between requested cleanup list and current unindexable partitions: [64]">>}},
        {gen_server,call,
        ['capi_set_view_manager-default',

        {set_vbucket_states, [active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,replica, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active], [undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined]}

        ,
        infinity]}},
        {gen_server,call,
        [

        {'janitor_agent-default', 'n_0@10.17.2.163'}

        ,
        {if_rebalance,<0.3699.0>,
        {update_vbucket_state,64,replica,
        undefined,'n_1@10.17.2.163'}},
        60000]}}}}]},
        [

        {janitor_agent,bulk_set_vbucket_state,4}

        ,

        {ns_vbucket_mover, update_replication_post_move,3}

        ,

        {ns_vbucket_mover,handle_info,2}

        ,

        {gen_server,handle_msg,5}

        ,

        {proc_lib,init_p_do_apply,3}

        ]}

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Thanks Filipe. I think there's similar crash now: Event Module Code Server Node Time Shutting down bucket "default" on 'n_0@10.17.2.163' for server shutdown (repeated 1 times) ns_memcached002 n_0@10.17.2.163 14:29:18 - Thu Aug 9, 2012 Bucket "default" loaded on node 'n_0@10.17.2.163' in 0 seconds. ns_memcached001 n_0@10.17.2.163 14:29:03 - Thu Aug 9, 2012 Bucket "default" loaded on node 'n_0@10.17.2.163' in 1 seconds. ns_memcached001 n_0@10.17.2.163 14:29:01 - Thu Aug 9, 2012 Rebalance exited with reason {{bulk_set_vbucket_state_failed, [{'n_0@10.17.2.163', {'EXIT', {{{bad_return_value, {error, <<"Intersection between requested cleanup list and current unindexable partitions: [64] ">>}}, {gen_server,call, ['capi_set_view_manager-default', {set_vbucket_states, [active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,replica, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active,active,active,active,active, active], [undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined,undefined,undefined, undefined]} , infinity]}}, {gen_server,call, [ {'janitor_agent-default', 'n_0@10.17.2.163'} , {if_rebalance,<0.3699.0>, {update_vbucket_state,64,replica, undefined,'n_1@10.17.2.163'}}, 60000]}}}}]}, [ {janitor_agent,bulk_set_vbucket_state,4} , {ns_vbucket_mover, update_replication_post_move,3} , {ns_vbucket_mover,handle_info,2} , {gen_server,handle_msg,5} , {proc_lib,init_p_do_apply,3} ]}
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Perhaps that's because I'm starting cleanup before unpausing vbucket.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Perhaps that's because I'm starting cleanup before unpausing vbucket.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Indeed. I've started unpausing before removing vbucket and it seems to work

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Indeed. I've started unpausing before removing vbucket and it seems to work
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Fixed now. thanks

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Fixed now. thanks

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            alkondratenko Aleksey Kondratenko (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes