Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-7702

Bad index state transition, regression added by MB-7522 (was: Queries fail with error {"error":"error","reason":"inconsistent_state"} during rebalance out)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • 2.0.1
    • 2.0.1
    • view-engine
    • Security Level: Public
    • centos 64, 2.0.1-150-rel

    Description

      build

      http://qa.hq.northscale.net/view/2.0.1/job/ubuntu-64-2.0-new-rebalance-tests-P0/342/consoleFull
      ./testrunner -i /tmp/rebalance-tests.ini get-logs=True,wait_timeout=180,GROUP=P0,get-cbcollect-info=True -t rebalance.rebalanceout.RebalanceOutTests.incremental_rebalance_out_with_queries,blob_generator=False,items=100000,GROUP=OUT;P0;FROM_2_0

      default bucket with 100K items, 1 ddoc with 5 views like as

      <<"{\"views\":{\"default_view0\":{\"map\":\"function (doc)

      { emit(doc._id, doc);}

      \"},\"default_view1\":{\"map\":\"function (doc)

      {\\n emit(doc._id, doc);\\n}

      \"}}}">>,
      0,false,[]}:

      test logs:
      [2013-02-06 13:49:39,877] - [rest_client:804] INFO - rebalance params : password=password&ejectedNodes=ns_1%4010.3.121.64%2Cns_1%4010.3.121.63&user=Administrator&knownNodes=ns_1%4010.5.2.13%2Cns_1%4010.3.121.63%2Cns_1%4010.5.2.14%2Cns_1%4010.3.121.64%2Cns_1%4010.5.2.15
      [2013-02-06 13:49:39,896] - [rest_client:808] INFO - rebalance operation started
      [2013-02-06 13:49:39,925] - [rest_client:905] INFO - rebalance percentage : 0 %
      [2013-02-06 13:49:49,945] - [rest_client:905] INFO - rebalance percentage : 5.67178565869 %
      [2013-02-06 13:49:59,964] - [rest_client:905] INFO - rebalance percentage : 13.571628723 %
      [2013-02-06 13:50:09,984] - [rest_client:905] INFO - rebalance percentage : 20.687479226 %
      [2013-02-06 13:50:16,019] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:50:39,620] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:51:03,456] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:51:23,628] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:51:54,119] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:52:22,573] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:52:43,810] - [task:1323] INFO - (100000 rows) expected, (96507 rows) returned
      [2013-02-06 13:52:43,902] - [task:308] INFO - rebalancing was completed with progress: 100% in 184.006303787 sec
      [2013-02-06 13:52:43,917] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:54:33,435] - [rest_client:578] ERROR - http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view1?connectionTimeout=60000&full_set=true&stale=update_after error 500 reason: error

      {"error":"error","reason":"inconsistent_state"}

      [('/usr/lib/python2.6/threading.py', 504, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.6/threading.py', 532, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 55, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 1344, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
      Wed Feb 6 13:54:33 2013
      [2013-02-06 13:54:33,475] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:56:23,022] - [rest_client:578] ERROR - http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view2?connectionTimeout=60000&full_set=true&stale=update_after error 500 reason: error

      {"error":"error","reason":"inconsistent_state"}

      [('/usr/lib/python2.6/threading.py', 504, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.6/threading.py', 532, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 55, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 1344, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
      Wed Feb 6 13:56:23 2013
      [2013-02-06 13:56:23,041] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 13:58:12,679] - [rest_client:578] ERROR - http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view3?connectionTimeout=60000&full_set=true&stale=update_after error 500 reason: error

      {"error":"error","reason":"inconsistent_state"}

      [('/usr/lib/python2.6/threading.py', 504, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.6/threading.py', 532, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 55, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 1344, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
      Wed Feb 6 13:58:12 2013
      [2013-02-06 13:58:12,697] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 14:00:02,082] - [rest_client:578] ERROR - http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view4?connectionTimeout=60000&full_set=true&stale=update_after error 500 reason: error

      {"error":"error","reason":"inconsistent_state"}

      [('/usr/lib/python2.6/threading.py', 504, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.6/threading.py', 532, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 55, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 1344, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
      Wed Feb 6 14:00:02 2013
      [2013-02-06 14:00:02,100] - [rest_client:329] INFO - index query url: http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after
      [2013-02-06 14:01:51,871] - [rest_client:578] ERROR - http://10.5.2.13:8092/default/_design/dev_ddoc1/_view/default_view0?connectionTimeout=60000&full_set=true&stale=update_after error 500 reason: error

      {"error":"error","reason":"inconsistent_state"}

      ERROR
      [('/usr/lib/python2.6/threading.py', 504, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.6/threading.py', 532, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 55, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 1344, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 263, 'set_exception', 'print traceback.extract_stack()')]
      Wed Feb 6 14:01:51 2013
      [('testrunner', 325, '<module>', 'result = unittest.TextTestRunner(verbosity=2).run(suite)'), ('/usr/lib/python2.6/unittest.py', 753, 'run', 'test(result)'), ('/usr/lib/python2.6/unittest.py', 464, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.6/unittest.py', 460, 'run', 'test(result)'), ('/usr/lib/python2.6/unittest.py', 300, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.6/unittest.py', 279, 'run', 'testMethod()'), ('pytests/rebalance/rebalanceout.py', 242, 'incremental_rebalance_out_with_queries', 'self.perform_verify_queries(num_views, prefix, ddoc_name, query, wait_time=timeout, expected_rows=expected_rows)'), ('pytests/basetestcase.py', 475, 'perform_verify_queries', 'task.result(wait_time)'), ('lib/tasks/future.py', 159, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 110, '__get_result', 'print traceback.extract_stack()')]
      Error occured querying view default_view0:

      {"error":"error","reason":"inconsistent_state"}

      ns logs:

      [couchdb:error,2013-02-06T13:54:33.433,ns_1@10.5.2.13:<0.18701.23>:couch_log:error:42]Uncaught error in HTTP request: {throw,{error,inconsistent_state}}

      Stacktrace: [

      {capi_view,design_doc_view_loop,6}

      ,

      {couch_httpd,handle_request,6}

      ,

      {mochiweb_http,headers,5}

      ,

      {proc_lib,init_p_do_apply,3}

      ]
      [views:debug,2013-02-06T13:54:33.949,ns_1@10.5.2.13:<0.10533.24>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.
      [views:debug,2013-02-06T13:54:39.425,ns_1@10.5.2.13:<0.10533.24>:capi_view:design_doc_view_loop:56]Got `set_view_outdated` error. Retrying.

      Filipe, could you look at the issue if it's the same as was in MB-6394, MB-6049,MB-6736
      please, reasign to Alk, if this is his area

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            andreibaranouski Andrei Baranouski
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty