Loading...

Details

Type: Bug
Resolution: Duplicate
Priority: Major
Fix Version/s: None
Affects Version/s: 7.6.0
Component/s: memcached
Labels:
Environment:
Couchbase Community Edition build 7.6.0-1811
Operating System : CentOS Linux 7 (Core)

Triage:
Untriaged
Operating System:
Centos 64-bit
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://cb-engineering.s3.amazonaws.com/MB-59736/collectinfo-2023-11-21T065023-ns_1@cb.local.zip
Story Points:
0
Is this a Regression?:
Unknown

Description

Steps to reproduce

1. Installed community edition on node using the command

$yum localinstall -y /tmp/couchbase-server-community-7.6.0-1813-linux.x86_64.rpm > /dev/null && echo 1 || echo 0

2. Attempted to connect to /pools/default endpoint

GET http://172.23.107.59:8091/pools/default body:  headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'}

error: 500 reason: unknown b'["Unexpected server error, request logged."]

Observing a lot of 500 code on ns_server.http_access.log

$grep "500 44" -m 50 ns_server.http_access.log

127.0.0.1 - - [20/Nov/2023:22:40:51 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - - 3536

127.0.0.1 - - [20/Nov/2023:22:40:55 -0800] "GET /pools/default HTTP/1.1" 500 44 - "CB-SearchService" 77

127.0.0.1 - - [20/Nov/2023:22:40:55 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 67

127.0.0.1 - - [20/Nov/2023:22:40:55 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 3296

127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1" 3710

127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 2035

127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3356

172.23.96.248 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default HTTP/1.1" 500 44 - "Python-httplib2/0.13.1 (gzip)" 119

127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 3044

127.0.0.1 - - [20/Nov/2023:22:41:00 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4037

127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 4079

127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4080

127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4081

127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2082

127.0.0.1 - - [20/Nov/2023:22:41:04 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 81

127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4011

127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3550

127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2013

127.0.0.1 - - [20/Nov/2023:22:41:08 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2015

127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2016

127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4015

127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4016

127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 33

127.0.0.1 - - [20/Nov/2023:22:41:12 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2017

127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3557

127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2016

127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4017

127.0.0.1 - - [20/Nov/2023:22:41:16 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2014

127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2015

127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2015

127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4017

127.0.0.1 - - [20/Nov/2023:22:41:20 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4018

127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4019

127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3558

127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2018

127.0.0.1 - - [20/Nov/2023:22:41:24 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2019

127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4018

127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2019

127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4021

127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2020

127.0.0.1 - - [20/Nov/2023:22:41:28 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "couchbase-goxdcr/7.6.0" 82

127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4023

127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3567

127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2022

127.0.0.1 - - [20/Nov/2023:22:41:32 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2021

127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /poolsStreaming/default HTTP/1.1" 500 44 - "CB-SearchService" 4015

127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-Indexer::refreshSecurityContextOnTopology" 2013

127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 4013

127.0.0.1 - - [20/Nov/2023:22:41:36 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "Go-http-client/1.1-indexer-SingletonCICLMgr" 2016

127.0.0.1 - - [20/Nov/2023:22:41:40 -0800] "GET /pools/default?uuid=fb14e1a478df4d7ede3db3d872126762 HTTP/1.1" 500 44 - "query" 3562

Observing a lot of CRASH REPORTS and Server error during processing: ["web request failed" in ns_server.debug.logs

[error_logger:error,2023-11-20T22:40:51.682-08:00,ns_1@cb.local:menelaus_web_cache<0.499.0>:ale_error_logger_handler:do_log:101]=========================CRASH REPORT=========================  crasher:    initial call: work_queue:init/1    pid: <0.499.0>    registered_name: menelaus_web_cache    exception exit: {{{badmatch,{error,couldnt_connect_to_memcached}},                      [{ns_audit_cfg,notify_memcached,1,                                     [{file,"src/ns_audit_cfg.erl"},                                      {line,153}]},                       {ns_audit_cfg,handle_info,2,                                     [{file,"src/ns_audit_cfg.erl"},                                      {line,128}]},                       {gen_server,try_dispatch,4,                                   [{file,"gen_server.erl"},{line,1123}]},                       {gen_server,handle_msg,6,                                   [{file,"gen_server.erl"},{line,1200}]},                       {proc_lib,init_p_do_apply,3,                                 [{file,"proc_lib.erl"},{line,240}]}]},                     {gen_server,call,[ns_audit_cfg,get_uid]}}      in function  gen_server:call/2 (gen_server.erl, line 370)      in call from menelaus_web_pools:build_internal_params/1 (src/menelaus_web_pools.erl, line 316)      in call from menelaus_web_pools:do_build_pool_info/4 (src/menelaus_web_pools.erl, line 254)      in call from menelaus_web_pools:'-build_pool_info/6-fun-1-'/5 (src/menelaus_web_pools.erl, line 190)      in call from menelaus_web_cache:do_compute_with_expiration/3 (src/menelaus_web_cache.erl, line 95)      in call from work_queue:handle_call/3 (src/work_queue.erl, line 51)      in call from gen_server:try_handle_call/4 (gen_server.erl, line 1149)      in call from gen_server:handle_msg/6 (gen_server.erl, line 1178)    ancestors: [menelaus_sup,ns_server_sup,ns_server_nodes_sup,<0.290.0>,                  ns_server_cluster_sup,root_sup,<0.155.0>]    message_queue_len: 0    messages: []    links: [<0.494.0>]    dictionary: []    trap_exit: false    status: running    heap_size: 6772    stack_size: 28    reductions: 44612  neighbours:

[ns_server:error,2023-11-20T22:40:51.683-08:00,ns_1@cb.local:<0.516.0>:menelaus_util:reply_server_error_before_close:216]Server error during processing: ["web request failed",                                 {path,"/poolsStreaming/default"},                                 {method,'GET'},                                 {type,exit},                                 {what,                                  {{{{badmatch,                                      {error,couldnt_connect_to_memcached}},                                     [{ns_audit_cfg,notify_memcached,1,                                       [{file,"src/ns_audit_cfg.erl"},                                        {line,153}]},                                      {ns_audit_cfg,handle_info,2,                                       [{file,"src/ns_audit_cfg.erl"},                                        {line,128}]},                                      {gen_server,try_dispatch,4,                                       [{file,"gen_server.erl"},{line,1123}]},                                      {gen_server,handle_msg,6,                                       [{file,"gen_server.erl"},{line,1200}]},                                      {proc_lib,init_p_do_apply,3,                                       [{file,"proc_lib.erl"},{line,240}]}]},                                    {gen_server,call,[ns_audit_cfg,get_uid]}},                                   {gen_server,call,                                    [menelaus_web_cache,                                     #Fun<menelaus_web_cache.2.98503217>,                                     infinity]}}},                                 {trace,                                  [{gen_server,call,3,                                    [{file,"gen_server.erl"},{line,385}]},                                   {menelaus_web_pools,pool_info,6,                                    [{file,"src/menelaus_web_pools.erl"},                                     {line,115}]},                                   {menelaus_util,                                    '-handle_streaming/2-fun-0-',3,                                    [{file,"src/menelaus_util.erl"},                                     {line,600}]},                                   {menelaus_util,streaming_inner,6,                                    [{file,"src/menelaus_util.erl"},                                     {line,669}]},                                   {menelaus_util,handle_streaming,7,                                    [{file,"src/menelaus_util.erl"},                                     {line,646}]},                                   {request_tracker,request,2,                                    [{file,"src/request_tracker.erl"},                                     {line,40}]},                                   {menelaus_util,handle_request,2,                                    [{file,"src/menelaus_util.erl"},                                     {line,227}]},                                   {mochiweb_http,headers,6,                                    [{file,                                      "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/mochiweb/mochiweb_http.erl"},                                     {line,153}]}]}]

Job link : https://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-1813/jenkins_logs/test_suite_executor/641003/

Attachments

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

[Community Edition] All requests to server returns status code 500 reason:"Unexpected server error, request logged."

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty