Details
-
Bug
-
Resolution: Fixed
-
Critical
-
7.6.0
-
Operating System : Microsoft WIndows 11
Couchbase Enterprise Edition build 7.6.0-1899
-
Untriaged
-
Windows 64-bit
-
-
0
-
Unknown
Description
Steps to repro
- Created a 6 node cluster
- Created 2 buckets 'default' and 'standard_bucket'
- Used cbdocloader on the buckets
Observing a panic in the ns_server.query.log on 172.23.136.103
panic: runtime error: invalid memory address or nil pointer dereference[signal 0xc0000005 code=0x0 addr=0x0 pc=0x15c0dbe] |
goroutine 1432 [running]:github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager.(*processManager).stopProcess(0xc0006b58c0?) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager/process_manager_impl.go:201 +0x3egithub.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager.(*processManager).createAndSendMessage(0xc000001200, 0x40?, {0x332d080?, 0xc00112c2a0?}, {0x0?, 0x2f76a554801?}, {0x0?, 0x278cee0?}, {0xc001113b48, 0x1, ...}) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager/process_manager_impl.go:193 +0xdegithub.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager.(*processManager).UpdateConfigEvent(0xc0004d6ba0?, {0x332d080, 0xc00112c2a0}, 0x9, {0xc00097fef0?, 0xc001113bc8?, 0xbaf056?}) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager/process_manager_impl.go:148 +0x2fbgithub.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).setConfig(0xc0001983f0, {0xc0?, {0xc00097fef0?, 0x46563c0?, 0x0?}}) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:328 +0x1e3github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init(0xc0001983f0) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:166 +0x1a8github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).respawnProcess(0xc0001983f0) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:425 +0x118github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).receiveResponseMsg(0xc0001983f0) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:377 +0x232created by github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init in goroutine 932 C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:161 +0xc7 |
Observing n1ql service crash in ns_server.debug.log soon after
[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_info:262]Observed json rpc process {rpc_process,"cbq-engine-cbauth",internal, #Ref<0.3821750981.1062993923.30593>,undefined, 2525438} died with reason shutdown[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29162.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29098.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29138.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {chronicle_compat_event_manager,<0.29136.6>} exited with reason shutdown[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29143.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29101.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29161.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29102.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.066-08:00,ns_1@172.23.136.103:<0.29189.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29127.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.066-08:00,ns_1@172.23.136.103:<0.29198.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29126.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.067-08:00,ns_1@172.23.136.103:<0.29203.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29135.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.067-08:00,ns_1@172.23.136.103:<0.29202.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29140.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.067-08:00,ns_1@172.23.136.103:<0.29204.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29139.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:json_rpc_connection-cbq-engine-service_api<0.29327.6>:json_rpc_connection:handle_info:142]Socket closed[ns_server:debug,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:<0.29329.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {chronicle_compat_event_manager,<0.29327.6>} exited with reason shutdown[ns_server:error,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:service_agent:handle_info:329]Lost json rpc connection for service n1ql, reason shutdown. Terminating.[ns_server:error,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:service_agent:terminate:350]Terminating abnormally[error_logger:error,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:ale_error_logger_handler:do_log:101]=========================ERROR REPORT=========================** Generic server 'service_agent-n1ql' terminating ** Last message in was {'DOWN',#Ref<0.3821750981.1062993923.31132>,process, <0.29327.6>,shutdown}** When Server state == {state,n1ql, {dict,2,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[], [[{node,'ns_1@172.23.136.103'}| <<"34f66682616ce6cd0dfe105dd712e77c">>]], [],[],[],[],[], [[{uuid,<<"34f66682616ce6cd0dfe105dd712e77c">>}| 'ns_1@172.23.136.103']], []}}}, undefined,undefined,undefined,undefined,undefined, undefined,undefined, {<<"AAAAAAAAAAI=">>,[]}, {<<"AAAAAAAAAAI=">>, {topology, ['ns_1@172.23.136.103'], [<<"34f66682616ce6cd0dfe105dd712e77c">>], true,[]}}, <0.29644.6>,<0.29645.6>,rebalance}** Reason for termination ==** {lost_connection,{'ns_1@172.23.136.103',shutdown}} |
[ns_server:debug,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:<0.30910.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.30795.6>} exited with reason normal[error_logger:error,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:ale_error_logger_handler:do_log:101]=========================CRASH REPORT========================= crasher: initial call: service_agent:init/1 pid: <0.28887.6> registered_name: 'service_agent-n1ql' exception exit: {lost_connection,{'ns_1@172.23.136.103',shutdown}} in function gen_server:handle_common_reply/8 (gen_server.erl, line 1241) ancestors: [service_agent_children_sup,service_agent_sup,ns_server_sup, ns_server_nodes_sup,<0.27670.6>,ns_server_cluster_sup, root_sup,<0.152.0>] message_queue_len: 2 messages: [{'EXIT',<0.29644.6>, {shutdown, {gen_server,call, [<0.29327.6>, {call,"ServiceAPI.GetTaskList", #Fun<json_rpc_connection.0.36915653>, #{timeout => 60000}}, 60000]}}}, {'EXIT',<0.29645.6>, {shutdown, {gen_server,call, [<0.29327.6>, {call,"ServiceAPI.GetCurrentTopology", #Fun<json_rpc_connection.0.36915653>, #{timeout => 60000}}, 60000]}}}] links: [<0.28889.6>,<0.28076.6>] dictionary: [] trap_exit: true status: running heap_size: 6772 stack_size: 28 reductions: 29604 neighbours: |
[ns_server:debug,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:<0.28889.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.28887.6>} exited with reason {lost_connection, {'ns_1@172.23.136.103', shutdown}}[error_logger:error,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:service_agent_children_sup<0.28076.6>:ale_error_logger_handler:do_log:101]=========================SUPERVISOR REPORT========================= supervisor: {local,service_agent_children_sup} errorContext: child_terminated reason: {lost_connection,{'ns_1@172.23.136.103',shutdown}} offender: [{pid,<0.28887.6>}, {id,{service_agent,n1ql}}, {mfargs,{service_agent,start_link,[n1ql]}}, {restart_type,permanent}, {significant,false}, {shutdown,1000}, {child_type,worker}] |
[ns_server:debug,2023-12-09T16:18:40.070-08:00,ns_1@172.23.136.103:<0.30915.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.30805.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.070-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"goxdcr-cbauth",[{internal,true}],<0.28528.6>} needs_update[error_logger:info,2023-12-09T16:18:40.072-08:00,ns_1@172.23.136.103:service_agent_children_sup<0.28076.6>:ale_error_logger_handler:do_log:101]=========================PROGRESS REPORT========================= supervisor: {local,service_agent_children_sup} started: [{pid,<0.615.7>}, {id,{service_agent,n1ql}}, {mfargs,{service_agent,start_link,[n1ql]}}, {restart_type,permanent}, {significant,false}, {shutdown,1000}, {child_type,worker}] |
[ns_server:debug,2023-12-09T16:18:40.072-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"fts-cbauth",[{internal,true}],<0.28951.6>} needs_update[ns_server:debug,2023-12-09T16:18:40.080-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"index-cbauth",[{internal,true}],<0.28962.6>} needs_update[ns_server:debug,2023-12-09T16:18:40.085-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"projector-cbauth",[{internal,true}],<0.29213.6>} needs_update[user:info,2023-12-09T16:18:40.103-08:00,ns_1@172.23.136.103:<0.27897.6>:ns_log:consume_log:76]Service 'n1ql' exited with status 2. Restarting. Messages: C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:328 +0x1e3github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init(0xc0001983f0) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:166 +0x1a8github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).respawnProcess(0xc0001983f0) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:425 +0x118github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).receiveResponseMsg(0xc0001983f0) C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:377 +0x232created by github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init in goroutine 932 C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:161 +0xc7 |
|
Testrunner script to repro
./testrunner -i /tmp/testexec.3008.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False,get-cbcollect-info=True,get-cbcollect-info=True -t clitest.docloadertest.docloaderTests.docloader_test,memory_quota=80,filename=beer-sample,standard_buckets=1,GROUP=P0
Job name : windows11-os_certify-tools_1