Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-4750

rebalancing two buckets under light write load is stuck at 50% ( possibly second bucket is not being scheduled for rebalance )

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-developer-preview-3
    • Fix Version/s: 2.0-developer-preview-4
    • Component/s: couchbase-bucket, ns_server
    • Security Level: Public
    • Labels:
      None
    • Environment:
      build 631

      Description

      Ok, looking at the vbucekts per node, it seems that the "couchbase" bucket, which is no empty, not used, no load, is the one that got the rebalance stuck.

      This is a removal node that got stuck.
      There are no active vbuckets on the node I removed for the default-active bucket, which is running fine.

      Seems an ns_server issue.

      1. Screen Shot 2012-02-02 at 11.43.39 AM.png
        53 kB
      2. Screen Shot 2012-02-02 at 11.43.42 AM.png
        39 kB
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        steve Steve Yen added a comment -

        possible duplicate of MB-4752

        Show
        steve Steve Yen added a comment - possible duplicate of MB-4752
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Part here is due to:

        [error_logger:error] [2012-02-02 19:00:42] [ns_1@10.176.150.73:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server <0.8037.1> terminating

          • Last message in was {'EXIT',<0.8041.1>,
            {{compact_filter_error,
            {error,badarith},
            {part_id,dups},
            {cbitmask,0}},
            [{couch_set_view_compactor, '-compact_group/5-fun-1-',2},
            {couch_btree_copy,fold_copy,3},
            {couch_btree,stream_kv_node2,8},
            {couch_btree,stream_kp_node,7},
            {couch_btree,fold,4},
            {couch_btree_copy,copy,3},
            {couch_set_view_compactor,compact_view,5},
            {lists,mapfoldl,3}]}}
            ** When Server state == {file,<0.8038.1>,<0.8039.1>,1242101}
            ** Reason for termination ==
            ** {{compact_filter_error,{error,badarith}

            ,

            {part_id,dups}

            ,{cbitmask,0}},
            [

            {couch_set_view_compactor,'-compact_group/5-fun-1-',2}

            ,

            {couch_btree_copy,fold_copy,3}

            ,

            {couch_btree,stream_kv_node2,8}

            ,

            {couch_btree,stream_kp_node,7}

            ,

            {couch_btree,fold,4}

            ,

            {couch_btree_copy,copy,3}

            ,

            {couch_set_view_compactor,compact_view,5}

            ,

            {lists,mapfoldl,3}

            ]}

        addressed by WIP commit here: http://review.couchbase.org/13067

        Part is bad error handling. One case is fixed by this: http://review.couchbase.org/13072

        Assigning to Filipe and confirming couch set nature of this.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Part here is due to: [error_logger:error] [2012-02-02 19:00:42] [ns_1@10.176.150.73:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server <0.8037.1> terminating Last message in was {'EXIT',<0.8041.1>, {{compact_filter_error, {error,badarith}, {part_id,dups}, {cbitmask,0}}, [{couch_set_view_compactor, '-compact_group/5-fun-1-',2}, {couch_btree_copy,fold_copy,3}, {couch_btree,stream_kv_node2,8}, {couch_btree,stream_kp_node,7}, {couch_btree,fold,4}, {couch_btree_copy,copy,3}, {couch_set_view_compactor,compact_view,5}, {lists,mapfoldl,3}]}} ** When Server state == {file,<0.8038.1>,<0.8039.1>,1242101} ** Reason for termination == ** {{compact_filter_error,{error,badarith} , {part_id,dups} ,{cbitmask,0}}, [ {couch_set_view_compactor,'-compact_group/5-fun-1-',2} , {couch_btree_copy,fold_copy,3} , {couch_btree,stream_kv_node2,8} , {couch_btree,stream_kp_node,7} , {couch_btree,fold,4} , {couch_btree_copy,copy,3} , {couch_set_view_compactor,compact_view,5} , {lists,mapfoldl,3} ]} addressed by WIP commit here: http://review.couchbase.org/13067 Part is bad error handling. One case is fixed by this: http://review.couchbase.org/13072 Assigning to Filipe and confirming couch set nature of this.
        Hide
        filipe manana filipe manana added a comment -

        Duplicate of MB-4774 and MB-4732.
        Already fixed.

        Show
        filipe manana filipe manana added a comment - Duplicate of MB-4774 and MB-4732 . Already fixed.

          People

          • Assignee:
            FilipeManana Filipe Manana (Inactive)
            Reporter:
            farshid Farshid Ghods (Inactive)
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes