Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
None
-
Security Level: Public
-
None
-
Centos 5.9
Python 2.6.8
libcouchbase 2.1.3
Description
I send about 400 queries per second during cluster rebalance from 4 to 3 nodes.
Usually queries fail at the end of rebalance:
[08/Sep/2013 06:35:09] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:14] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:19] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:24] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:29] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:34] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:39] INFO - Rebalance progress: 98.6301317402 %
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%227f658f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227f658f%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%227eece5%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%227eece5%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22274893%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22274893%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2216fa16%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2216fa16%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22de8507%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22de8507%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2263906a%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2263906a%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22e8d498%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22e8d498%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d4950f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d4950f%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22918842%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22918842%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22a48888%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22a48888%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22563aa5%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22563aa5%22>>
[08/Sep/2013 06:35:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2261691f%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2261691f%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%228f69ca%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%228f69ca%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%229bc615%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%229bc615%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d6f243%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d6f243%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2268c4fd%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2268c4fd%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%223d8d81%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%223d8d81%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22d8a078%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22d8a078%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%221ed84b%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%221ed84b%22>>
[08/Sep/2013 06:35:43] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2296c892%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2296c892%22>>
[08/Sep/2013 06:35:44] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:35:49] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:35:54] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:35:59] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:04] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:09] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:14] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:19] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:24] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:29] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:34] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:39] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:44] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:49] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:54] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:36:59] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:04] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:09] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:14] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:19] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:24] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:29] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:34] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:39] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:44] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:49] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:54] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:37:59] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:04] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:09] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:14] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:19] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:24] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:29] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:34] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:39] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:44] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:49] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:54] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:38:59] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:04] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:09] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:14] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:19] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:24] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:29] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:34] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:39] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:44] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:49] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:54] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:39:59] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:04] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:09] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:14] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:19] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:24] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:30] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:35] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:40] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:45] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:50] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:40:55] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:00] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:05] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:10] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:15] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:20] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:25] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:30] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:35] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:40] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:45] INFO - Rebalance progress: 100.0 %
[08/Sep/2013 06:41:50] INFO - Rebalance successfully completed
Sometimes I keep receiving errors even after rebalance completion.
[26/Sep/2013 08:33:40] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22a1524e%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
{u'reason': u'Cannot execute view query since the node has no active vbuckets', u'error': u'no_active_vbuckets'}, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%22a1524e%22>>
[26/Sep/2013 08:33:40] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%2224a826%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=302, URL=_design/A/_view/id_by_city?limit=30&key=%2224a826%22>>
[26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%223dbeb3%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%223dbeb3%22>>
[26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%225cc1f0%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%225cc1f0%22>>
[26/Sep/2013 08:33:42] WARNING - <Key='_design/A/_view/id_by_city?limit=30&key=%22433480%22', HTTP Request failed. Examine 'objextra' for full result, C Source=(src/http.c,225), OBJ=HttpResult<RC=0x0, Value=
, HTTP=404, URL=_design/A/_view/id_by_city?limit=30&key=%22433480%22>>
In this case I receive 404 code after 302.