Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
7.6.0
-
Couchbase Community Edition build 7.6.0-1811
Operating System : CentOS Linux 7 (Core)
-
Untriaged
-
Centos 64-bit
-
0
-
Unknown
Description
Steps to reproduce
1. Installed community edition on node using the command
$yum localinstall -y /tmp/couchbase-server-community-7.6.0-1813-linux.x86_64.rpm > /dev/null && echo 1 || echo 0 |
|
1 |
2. Attempted to connect to /pools/default endpoint
GET http://172.23.107.59:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} |
|
error: 500 reason: unknown b'["Unexpected server error, request logged."] |
Observing a lot of 500 code on ns_server.http_access.log
$grep "500 44" -m 50 ns_server.http_access.log |
|
127.0.0.1 - - [20/Nov/2023:22:40:51 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - - 3536 |
127.0.0.1 - - [20/Nov/2023:22:40:55 -0800] "GET /pools/default HTTP/1.1" 500 44 - "CB-SearchService" 77 |
127.0.0.1 - - [20/Nov/2023:22:40:55 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 67 |
127.0.0.1 - - [20/Nov/2023:22:40:55 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 3296 |
127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1" 3710 |
127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 2035 |
127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3356 |
172.23.96.248 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default HTTP/1.1" 500 44 - "Python-httplib2/0.13.1 (gzip)" 119 |
127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 3044 |
127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4037 |
127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 4079 |
127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4080 |
127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4081 |
127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2082 |
127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 81 |
127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4011 |
127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3550 |
127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2013 |
127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2015 |
127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2016 |
127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4015 |
127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4016 |
127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 33 |
127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2017 |
127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3557 |
127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2016 |
127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4017 |
127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2014 |
127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2015 |
127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2015 |
127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4017 |
127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4018 |
127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4019 |
127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3558 |
127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2018 |
127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2019 |
127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4018 |
127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2019 |
127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4021 |
127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2020 |
127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 82 |
127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4023 |
127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3567 |
127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2022 |
127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2021 |
127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4015 |
127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2013 |
127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4013 |
127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2016 |
127.0.0.1 - - [20/Nov/2023:22:41:40 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3562 |
Observing a lot of CRASH REPORTS and Server error during processing: ["web request failed" in ns_server.debug.logs
[error_logger:error,2023-11-20T22:40:51.682-08:00,ns_1@cb.local:menelaus_web_cache<0.499.0>:ale_error_logger_handler:do_log:101]=========================CRASH REPORT========================= crasher: initial call: work_queue:init/1 pid: <0.499.0> registered_name: menelaus_web_cache exception exit: {{{badmatch,{error,couldnt_connect_to_memcached}}, [{ns_audit_cfg,notify_memcached,1, [{file,"src/ns_audit_cfg.erl"}, {line,153}]}, {ns_audit_cfg,handle_info,2, [{file,"src/ns_audit_cfg.erl"}, {line,128}]}, {gen_server,try_dispatch,4, [{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,240}]}]}, {gen_server,call,[ns_audit_cfg,get_uid]}} in function gen_server:call/2 (gen_server.erl, line 370) in call from menelaus_web_pools:build_internal_params/1 (src/menelaus_web_pools.erl, line 316) in call from menelaus_web_pools:do_build_pool_info/4 (src/menelaus_web_pools.erl, line 254) in call from menelaus_web_pools:'-build_pool_info/6-fun-1-'/5 (src/menelaus_web_pools.erl, line 190) in call from menelaus_web_cache:do_compute_with_expiration/3 (src/menelaus_web_cache.erl, line 95) in call from work_queue:handle_call/3 (src/work_queue.erl, line 51) in call from gen_server:try_handle_call/4 (gen_server.erl, line 1149) in call from gen_server:handle_msg/6 (gen_server.erl, line 1178) ancestors: [menelaus_sup,ns_server_sup,ns_server_nodes_sup,<0.290.0>, ns_server_cluster_sup,root_sup,<0.155.0>] message_queue_len: 0 messages: [] links: [<0.494.0>] dictionary: [] trap_exit: false status: running heap_size: 6772 stack_size: 28 reductions: 44612 neighbours: |
[ns_server:error,2023-11-20T22:40:51.683-08:00,ns_1@cb.local:<0.516.0>:menelaus_util:reply_server_error_before_close:216]Server error during processing: ["web request failed", {path,"/poolsStreaming/default"}, {method,'GET'}, {type,exit}, {what, {{{{badmatch, {error,couldnt_connect_to_memcached}}, [{ns_audit_cfg,notify_memcached,1, [{file,"src/ns_audit_cfg.erl"}, {line,153}]}, {ns_audit_cfg,handle_info,2, [{file,"src/ns_audit_cfg.erl"}, {line,128}]}, {gen_server,try_dispatch,4, [{file,"gen_server.erl"},{line,1123}]}, {gen_server,handle_msg,6, [{file,"gen_server.erl"},{line,1200}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,240}]}]}, {gen_server,call,[ns_audit_cfg,get_uid]}}, {gen_server,call, [menelaus_web_cache, #Fun<menelaus_web_cache.2.98503217>, infinity]}}}, {trace, [{gen_server,call,3, [{file,"gen_server.erl"},{line,385}]}, {menelaus_web_pools,pool_info,6, [{file,"src/menelaus_web_pools.erl"}, {line,115}]}, {menelaus_util, '-handle_streaming/2-fun-0-',3, [{file,"src/menelaus_util.erl"}, {line,600}]}, {menelaus_util,streaming_inner,6, [{file,"src/menelaus_util.erl"}, {line,669}]}, {menelaus_util,handle_streaming,7, [{file,"src/menelaus_util.erl"}, {line,646}]}, {request_tracker,request,2, [{file,"src/request_tracker.erl"}, {line,40}]}, {menelaus_util,handle_request,2, [{file,"src/menelaus_util.erl"}, {line,227}]}, {mochiweb_http,headers,6, [{file, "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"}, {line,153}]}]}] |