Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-6160

Rebalance failed initial call: ns_single_vbucket_mover:mover/6 pid: <0.17517.2> registered_name: [] exception error: {bulk_set_vbucket_state_failed,

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta
    • Fix Version/s: 2.0-beta
    • Component/s: ns_server
    • Security Level: Public
    • Labels:
      None

      Description

      Rebalance Failed on build-1554 for test:
      ./testrunner -i andrei.ini -t view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops,ddoc_ops=delete,test_with_view=True,num_ddocs=2,num_views_per_ddoc=3,items=2000

      [error_logger:error] [2012-08-08 12:54:03] [ns_1@10.5.2.11:error_logger:ale_error_logger_handler:log_report:72]
      =========================CRASH REPORT=========================
      crasher:
      initial call: ns_single_vbucket_mover:mover/6
      pid: <0.17517.2>
      registered_name: []
      exception error: {bulk_set_vbucket_state_failed,
      [{'ns_1@10.5.2.19',
      {'EXIT',
      {{{normal,
      {gen_server,call,
      [<20615.11732.0>,

      {set_state,[],"k", [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17, 18,19,20,21,22,23,24,25,26,27,28,29,30,31,32, 33,34,35,36,37,38,39,40,41,42,43,44,45,46,47, 48,49,50,51,52,53,54,55,56,57,58,59,60,61,62, 63,64,65,66,67,68,69,70,71,72,73,74,75,76,77, 78,79,80,81,82,83,84,85,86,87,88,89,90,91,92, 93,94,95,96,97,98,99,100,101,102,103,104,105, 106,108,109,110,111,112,113,114,115,116,117, 118,119,120,121,122,123,124,125,126,127]}

      ,
      infinity]}},
      {gen_server,call,

      1. 10.5.2.11-8091-diag.txt.gz
        3.49 MB
        Andrei Baranouski
      2. 10.5.2.13-8091-diag.txt.gz
        2.10 MB
        Andrei Baranouski
      3. 10.5.2.14-8091-diag.txt.gz
        950 kB
        Andrei Baranouski
      4. 10.5.2.15-8091-diag.txt.gz
        955 kB
        Andrei Baranouski
      5. 10.5.2.16-8091-diag.txt.gz
        954 kB
        Andrei Baranouski
      6. 10.5.2.18-8091-diag.txt.gz
        977 kB
        Andrei Baranouski
      7. 10.5.2.19-8091-diag.txt.gz
        1.73 MB
        Andrei Baranouski
      8. rebalance_with_ddoc_ops.log
        12 kB
        Andrei Baranouski
      No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Is that possible that you've deleted some design doc during rebalance?

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Is that possible that you've deleted some design doc during rebalance?
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        Yep, Alex

        The test performs the deletion of ddoc during rebalancing:
        ...
        [2012-08-08 12:54:16,531] - [task:682] INFO - view : views2 was created successfully in ddoc: dev_ddoc1
        [2012-08-08 12:54:16,543] - [createdeleteview:106] INFO - Processing Delete DDoc Operation On Bucket default
        [2012-08-08 12:54:19,616] - [rest_client:247] INFO - index query url: {http://10.5.2.11:8092//default/_design/dev_ddoc1/_view/views0?stale=ok
        [2012-08-08 12:54:19,627] - [rest_client:566] ERROR - {http://10.5.2.11:8092//default/_design/dev_ddoc1/_view/views0?stale=ok error 500 reason: error

        {"error":"error","reason":"View `views0` not defined in local design document `_design/dev_ddoc1`."}

        [2012-08-08 12:54:19,628] - [task:782] INFO - view : views0 was successfully deleted in ddoc: dev_ddoc1
        ERROR
        ...

        ERROR: rebalance_in_and_out_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests)
        ----------------------------------------------------------------------
        Traceback (most recent call last):
        File "pytests/view/createdeleteview.py", line 426, in rebalance_in_and_out_with_ddoc_ops
        rebalance.result()
        File "lib/tasks/future.py", line 149, in result
        return self.__get_result()
        File "lib/tasks/future.py", line 108, in __get_result
        raise self._exception
        RebalanceFailedException: Rebalance Failed:

        {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'}

        - rebalance failed

        ----------------------------------------------------------------------

        Show
        andreibaranouski Andrei Baranouski added a comment - Yep, Alex The test performs the deletion of ddoc during rebalancing: ... [2012-08-08 12:54:16,531] - [task:682] INFO - view : views2 was created successfully in ddoc: dev_ddoc1 [2012-08-08 12:54:16,543] - [createdeleteview:106] INFO - Processing Delete DDoc Operation On Bucket default [2012-08-08 12:54:19,616] - [rest_client:247] INFO - index query url: { http://10.5.2.11:8092//default/_design/dev_ddoc1/_view/views0?stale=ok [2012-08-08 12:54:19,627] - [rest_client:566] ERROR - { http://10.5.2.11:8092//default/_design/dev_ddoc1/_view/views0?stale=ok error 500 reason: error {"error":"error","reason":"View `views0` not defined in local design document `_design/dev_ddoc1`."} [2012-08-08 12:54:19,628] - [task:782] INFO - view : views0 was successfully deleted in ddoc: dev_ddoc1 ERROR ... ERROR: rebalance_in_and_out_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/view/createdeleteview.py", line 426, in rebalance_in_and_out_with_ddoc_ops rebalance.result() File "lib/tasks/future.py", line 149, in result return self.__get_result() File "lib/tasks/future.py", line 108, in __get_result raise self._exception RebalanceFailedException: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try rebalance again.'} - rebalance failed ----------------------------------------------------------------------
        Hide
        andreibaranouski Andrei Baranouski added a comment -

        execution_test_results

        Show
        andreibaranouski Andrei Baranouski added a comment - execution_test_results
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Ok thanks. I think I'll need better strategy on handling design doc deletions while doing work in capi_set_view_manager.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Ok thanks. I think I'll need better strategy on handling design doc deletions while doing work in capi_set_view_manager.
        Hide
        FilipeManana Filipe Manana (Inactive) added a comment -

        Same could happen if ddoc is updated with different map or reduce function.

        Show
        FilipeManana Filipe Manana (Inactive) added a comment - Same could happen if ddoc is updated with different map or reduce function.
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Thanks, Filipe. I plan to block ddoc updates/deletions for duration of capi_set_view_manager calls into set_views code.

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Thanks, Filipe. I plan to block ddoc updates/deletions for duration of capi_set_view_manager calls into set_views code.
        Hide
        thuan Thuan Nguyen added a comment -

        Integrated in github-ns-server-2-0 #435 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/435/)
        MB-6160: apply index config changes on ddoc replication process (Revision 35c3725248848b4fa33bfcddd79ff682c2098f8a)

        Result = SUCCESS
        Aliaksey Kandratsenka :
        Files :

        • src/menelaus_stats.erl
        • src/couch_stats_reader.erl
        • src/janitor_agent.erl
        • src/compaction_daemon.erl
        • src/cb_generic_replication_srv.erl
        • src/set_view_update_daemon.erl
        • src/menelaus_web_buckets.erl
        • src/capi_set_view_manager.erl
        • src/capi_ddoc_replication_srv.erl
        Show
        thuan Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #435 (See http://qa.hq.northscale.net/job/github-ns-server-2-0/435/ ) MB-6160 : apply index config changes on ddoc replication process (Revision 35c3725248848b4fa33bfcddd79ff682c2098f8a) Result = SUCCESS Aliaksey Kandratsenka : Files : src/menelaus_stats.erl src/couch_stats_reader.erl src/janitor_agent.erl src/compaction_daemon.erl src/cb_generic_replication_srv.erl src/set_view_update_daemon.erl src/menelaus_web_buckets.erl src/capi_set_view_manager.erl src/capi_ddoc_replication_srv.erl
        Hide
        alkondratenko Aleksey Kondratenko (Inactive) added a comment -

        Fixed recently by doing all set views call on ddoc replication process, serializing things

        Show
        alkondratenko Aleksey Kondratenko (Inactive) added a comment - Fixed recently by doing all set views call on ddoc replication process, serializing things

          People

          • Assignee:
            alkondratenko Aleksey Kondratenko (Inactive)
            Reporter:
            andreibaranouski Andrei Baranouski
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Gerrit Reviews

              There are no open Gerrit changes