Details
-
Bug
-
Resolution: Duplicate
-
Major
-
Cheshire-Cat
-
Couchbase EE 7.0.0-4603
-
Untriaged
-
Centos 64-bit
-
-
1
-
No
Description
Summary:
Ensure manifest api returns generic error "500 stats error: Unexpected server error, request logged" during rebalance-in
Steps to Repro
1. 3 init nodes
+----------------+----------+-----------------------+---------------+--------------+
|
| Nodes | Services | Version | CPU | Status |
|
+----------------+----------+-----------------------+---------------+--------------+
|
| 172.23.105.215 | kv | 7.0.0-4603-enterprise | 2.66800906116 | Cluster node |
|
| 172.23.105.217 | None | | | <--- IN --- |
|
| 172.23.105.219 | None | | | <--- IN --- |
|
+----------------+----------+-----------------------+---------------+--------------+
|
2. Create buckets
+---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+
|
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
|
+---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+
|
| bucket1 | couchbase | 3 | none | 0 | 3000 | 629145600 | 151372904 | 325644284 |
|
| bucket2 | ephemeral | 3 | none | 0 | 3000 | 629145600 | 310597160 | 102 |
|
| default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 571091272 | 592701269 |
|
+---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+
|
3. Rebalance-in and create collections and use manifest api to pass the uid of the created collection
2021-03-03 21:55:06,019 | test | INFO | pool-3-thread-23 | [table_view:display:72] Rebalance Overview
----------------------------------------------------------------------
| Nodes | Services | Version | CPU | Status |
----------------------------------------------------------------------
| 172.23.105.215 | kv | 7.0.0-4603-enterprise | 30.888719124 | Cluster node |
| 172.23.105.217 | kv | 7.0.0-4603-enterprise | 29.2751403777 | Cluster node |
| 172.23.105.219 | kv | 7.0.0-4603-enterprise | 28.5604900459 | Cluster node |
| 172.23.105.220 | None | <--- IN — | ||
| 172.23.106.237 | None | <--- IN — |
----------------------------------------------------------------------
2021-03-03 21:55:06,059 | test | INFO | pool-3-thread-23 | [task:check:331] Rebalance - status: running, progress: 0
|
2021-03-03 21:55:07,322 | infra | ERROR | ThreadPoolExecutor-30_0 | [Rest_Connection:_http_request:253] POST http://172.23.105.215:8091/pools/default/buckets/bucket1/scopes/@ensureManifest/db body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password
|
2021-03-03 21:55:07,415 | infra | ERROR | ThreadPoolExecutor-30_1 | [Rest_Connection:_http_request:253] POST http://172.23.105.215:8091/pools/default/buckets/bucket1/scopes/@ensureManifest/dc body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Content-Type': 'application/x-www-form-urlencoded'} error: 500 reason: unknown ["Unexpected server error, request logged."] auth: Administrator:password
|
checking .215 error.log
[ns_server:error,2021-03-03T21:55:07.407-08:00,ns_1@172.23.105.215:<0.6521.1>:menelaus_util:reply_server_error:207]Server error during processing: ["web request failed",
|
{path,
|
"/pools/default/buckets/bucket1/scopes/@ensureManifest/dc"},
|
{method,'POST'},
|
{type,error},
|
{what,
|
{badmatch,
|
{[{'ns_1@172.23.105.219',ok},
|
{'ns_1@172.23.105.217',ok},
|
{'ns_1@172.23.105.215',ok}],
|
[{'ns_1@172.23.105.220',
|
{'EXIT',
|
{function_clause,
|
[{collections,
|
convert_uid_from_memcached,
|
[{error,
|
{select_bucket_failed,
|
{memcached_error,key_enoent,
|
undefined}}}],
|
[{file,"src/collections.erl"},
|
{line,167}]},
|
{collections,wait_for_manifest_uid,
|
3,
|
[{file,"src/collections.erl"},
|
{line,657}]},
|
{async,'-async_init/4-fun-1-',3,
|
[{file,"src/async.erl"},
|
{line,197}]}]}}},
|
{'ns_1@172.23.106.237',
|
{'EXIT',
|
{function_clause,
|
[{collections,
|
convert_uid_from_memcached,
|
[{error,
|
{select_bucket_failed,
|
Logs attached (But unfortunately did not receive snapshot link for some reason )