Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: library
    • Security Level: Public
    • Labels:
      None
    • Environment:
      Centos 5.9
      Python 2.6.8
      libcouchbase 2.1.3

      Description

      I send about 400 queries per second during cluster rebalance from 4 to 3 nodes.

      Usually queries fail at the end of rebalance:

      [08/Sep/2013 06:35:09] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:14] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:19] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:24] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:29] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:34] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:39] INFO - Rebalance progress: 98.6301317402 %
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%227f658f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227f658f%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%227eece5%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227eece5%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22274893%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22274893%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2216fa16%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2216fa16%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22de8507%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22de8507%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2263906a%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2263906a%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22e8d498%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22e8d498%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d4950f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d4950f%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22918842%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22918842%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22a48888%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22a48888%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22563aa5%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22563aa5%22>>
      [08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2261691f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2261691f%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%228f69ca%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%228f69ca%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%229bc615%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%229bc615%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d6f243%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d6f243%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2268c4fd%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2268c4fd%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%223d8d81%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%223d8d81%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d8a078%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d8a078%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%221ed84b%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%221ed84b%22>>
      [08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2296c892%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2296c892%22>>
      [08/Sep/2013 06:35:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:35:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:35:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:35:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:36:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:37:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:38:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:29] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:34] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:39] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:44] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:49] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:54] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:39:59] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:04] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:09] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:14] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:19] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:24] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:30] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:35] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:40] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:45] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:50] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:40:55] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:00] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:05] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:10] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:15] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:20] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:25] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:30] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:35] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:40] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:45] INFO - Rebalance progress: 100.0 %
      [08/Sep/2013 06:41:50] INFO - Rebalance successfully completed

      Sometimes I keep receiving errors even after rebalance completion.

      [26/Sep/2013 08:33:40] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22a1524e%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22a1524e%22>>
      [26/Sep/2013 08:33:40] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2224a826%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}

      , HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2224a826%22>>
      [26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%223dbeb3%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'missing', u'error': u'not_found'}

      , HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%223dbeb3%22>>
      [26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%225cc1f0%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'missing', u'error': u'not_found'}

      , HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%225cc1f0%22>>
      [26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22433480%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=

      {u'reason': u'missing', u'error': u'not_found'}

      , HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%22433480%22>>

      In this case I receive 404 code after 302.

        Issue Links

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

          Hide
          ingenthr Matt Ingenthron added a comment -

          This is one of those situations we've worked around in the Java client with some retries, but it's still problematic. I don't feel very good retrying on 404s.

          Show
          ingenthr Matt Ingenthron added a comment - This is one of those situations we've worked around in the Java client with some retries, but it's still problematic. I don't feel very good retrying on 404s.
          Hide
          avsej Sergey Avseyev added a comment -

          The libcouchbase fix in the gerrit already. See related ticket

          Show
          avsej Sergey Avseyev added a comment - The libcouchbase fix in the gerrit already. See related ticket
          Hide
          pavelpaulau Pavel Paulau added a comment - - edited

          Observing other problem after libcouchbase upgrade:

          CouchbaseError: <Key='_design/A/_view/id_by_city?limit=30&key=%227b3e7f%22', RC=0x29[Maximum allowed number redirects reached. See lcb_cntl(3) manpage for LCB_CNTL_MAX_REDIRECTS option to get or set this limit.], HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,241), OBJ=HttpResult<RC=0x29[Maximum allowed number redirects reached. See lcb_cntl(3) manpage for LCB_CNTL_MAX_REDIRECTS option to get or set this limit.], Value=[], HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227b3e7f%22>

          I'm probably going to try LCB_CNTL_MAX_REDIRECTS = -1.

          Show
          pavelpaulau Pavel Paulau added a comment - - edited Observing other problem after libcouchbase upgrade: CouchbaseError: <Key='_design/A/_view/id_by_city?limit=30&key=%227b3e7f%22', RC=0x29 [Maximum allowed number redirects reached. See lcb_cntl(3) manpage for LCB_CNTL_MAX_REDIRECTS option to get or set this limit.] , HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,241), OBJ=HttpResult<RC=0x29 [Maximum allowed number redirects reached. See lcb_cntl(3) manpage for LCB_CNTL_MAX_REDIRECTS option to get or set this limit.] , Value=[], HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227b3e7f%22> I'm probably going to try LCB_CNTL_MAX_REDIRECTS = -1.
          Hide
          avsej Sergey Avseyev added a comment -

          Yeah, that's probably the feature. By default we allow three redirects. If you have seen other sane/practical value, I think I can adjust defaults. I think setting it to -1 might be not the best solution, at least for defaults in libcouchbase

          Show
          avsej Sergey Avseyev added a comment - Yeah, that's probably the feature. By default we allow three redirects. If you have seen other sane/practical value, I think I can adjust defaults. I think setting it to -1 might be not the best solution, at least for defaults in libcouchbase
          Hide
          ingenthr Matt Ingenthron added a comment -

          Pavel: I noticed you just marked this as fixed, so it looks like you're in agreement that we shouldn't need to handle more than 3 redirects? Would like your feedback based on your testing as this is still an area we're working through.

          Show
          ingenthr Matt Ingenthron added a comment - Pavel: I noticed you just marked this as fixed, so it looks like you're in agreement that we shouldn't need to handle more than 3 redirects? Would like your feedback based on your testing as this is still an area we're working through.
          Hide
          pavelpaulau Pavel Paulau added a comment -

          I'm just going to create a separate issue.)

          Show
          pavelpaulau Pavel Paulau added a comment - I'm just going to create a separate issue.)
          Hide
          ingenthr Matt Ingenthron added a comment -

          Better is to move this one to CCBC, which I already did. That way, we don't lose the history.

          Show
          ingenthr Matt Ingenthron added a comment - Better is to move this one to CCBC, which I already did. That way, we don't lose the history.
          Hide
          pavelpaulau Pavel Paulau added a comment -

          Oh, too late.) But feel free to close CCBC-295 and continue discussion here.

          Show
          pavelpaulau Pavel Paulau added a comment - Oh, too late.) But feel free to close CCBC-295 and continue discussion here.

            People

            • Assignee:
              mnunberg Mark Nunberg
              Reporter:
              pavelpaulau Pavel Paulau
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Gerrit Reviews

                There are no open Gerrit changes