Uploaded image for project: 'Couchbase C client library libcouchbase'
  1. Couchbase C client library libcouchbase
  2. CCBC-294

Cannot execute view query at the end of rebalance out

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • None
    • None
    • library
    • Security Level: Public
    • None
    • Centos 5.9
      Python 2.6.8
      libcouchbase 2.1.3

    Description

      I send about 400 queries per second during cluster rebalance from 4 to 3 nodes.

      Usually queries fail at the end of rebalance:

      [08/Sep/2013 06:35:09] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:14] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:19] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:24] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:29] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:34] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:39] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%227f658f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227f658f%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%227eece5%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227eece5%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22274893%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22274893%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2216fa16%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2216fa16%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22de8507%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22de8507%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2263906a%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2263906a%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22e8d498%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22e8d498%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d4950f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d4950f%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22918842%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22918842%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22a48888%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22a48888%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22563aa5%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22563aa5%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2261691f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2261691f%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%228f69ca%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%228f69ca%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%229bc615%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%229bc615%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d6f243%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d6f243%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2268c4fd%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2268c4fd%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%223d8d81%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%223d8d81%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d8a078%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d8a078%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%221ed84b%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%221ed84b%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2296c892%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2296c892%22>>
      [08/Sep/2013 06:35:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:35:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:35:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:35:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:30] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:35] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:40] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:45] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:50] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:55] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:00] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:05] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:10] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:15] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:20] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:25] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:30] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:35] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:40] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:45] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:50] INFO - Rebalance successfully completed

      Sometimes I keep receiving errors even after rebalance completion.

      [26/Sep/2013 08:33:40] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22a1524e%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22a1524e%22>>
      [26/Sep/2013 08:33:40] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2224a826%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2224a826%22>>
      [26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%223dbeb3%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'missing', u'error': u'not_found'}

      , HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%223dbeb3%22>>
      [26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%225cc1f0%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'missing', u'error': u'not_found'}

      , HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%225cc1f0%22>>
      [26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22433480%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'missing', u'error': u'not_found'}

      , HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%22433480%22>>

      In this case I receive 404 code after 302.

      Attachments

        Issue Links

          depends on

          New Feature - A new feature of the product, which has yet to be developed. CCBC-169 Handle 30x HTTP response code when chosen node doesn't have active vbuckets

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. CCBC-295 Default LCB_CNTL_MAX_REDIRECTS may be too low for view queries during rebalance

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mnunberg Mark Nunberg (Inactive)
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty