Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60057

[Query][Windows] panic: runtime error: invalid memory address or nil pointer dereference

    XMLWordPrintable

Details

    Description

      Steps to repro

      1. Created a 6 node cluster
      2. Created 2 buckets 'default' and 'standard_bucket'
      3. Used cbdocloader on the buckets

      Observing a panic in the ns_server.query.log on 172.23.136.103

      panic: runtime error: invalid memory address or nil pointer dereference[signal 0xc0000005 code=0x0 addr=0x0 pc=0x15c0dbe]
      goroutine 1432 [running]:github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager.(*processManager).stopProcess(0xc0006b58c0?)    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager/process_manager_impl.go:201 +0x3egithub.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager.(*processManager).createAndSendMessage(0xc000001200, 0x40?, {0x332d080?, 0xc00112c2a0?}, {0x0?, 0x2f76a554801?}, {0x0?, 0x278cee0?}, {0xc001113b48, 0x1, ...})    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager/process_manager_impl.go:193 +0xdegithub.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager.(*processManager).UpdateConfigEvent(0xc0004d6ba0?, {0x332d080, 0xc00112c2a0}, 0x9, {0xc00097fef0?, 0xc001113bc8?, 0xbaf056?})    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_manager/process_manager_impl.go:148 +0x2fbgithub.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).setConfig(0xc0001983f0, {0xc0?, {0xc00097fef0?, 0x46563c0?, 0x0?}})    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:328 +0x1e3github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init(0xc0001983f0)    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:166 +0x1a8github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).respawnProcess(0xc0001983f0)    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:425 +0x118github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).receiveResponseMsg(0xc0001983f0)    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:377 +0x232created by github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init in goroutine 932    C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:161 +0xc7 

      Observing n1ql service crash in ns_server.debug.log soon after

      [ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_info:262]Observed json rpc process {rpc_process,"cbq-engine-cbauth",internal,                              #Ref<0.3821750981.1062993923.30593>,undefined,                              2525438} died with reason shutdown[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29162.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29098.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29138.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {chronicle_compat_event_manager,<0.29136.6>} exited with reason shutdown[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29143.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29101.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.065-08:00,ns_1@172.23.136.103:<0.29161.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29102.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.066-08:00,ns_1@172.23.136.103:<0.29189.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29127.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.066-08:00,ns_1@172.23.136.103:<0.29198.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29126.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.067-08:00,ns_1@172.23.136.103:<0.29203.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29135.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.067-08:00,ns_1@172.23.136.103:<0.29202.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29140.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.067-08:00,ns_1@172.23.136.103:<0.29204.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.29139.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:json_rpc_connection-cbq-engine-service_api<0.29327.6>:json_rpc_connection:handle_info:142]Socket closed[ns_server:debug,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:<0.29329.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {chronicle_compat_event_manager,<0.29327.6>} exited with reason shutdown[ns_server:error,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:service_agent:handle_info:329]Lost json rpc connection for service n1ql, reason shutdown. Terminating.[ns_server:error,2023-12-09T16:18:40.068-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:service_agent:terminate:350]Terminating abnormally[error_logger:error,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:ale_error_logger_handler:do_log:101]=========================ERROR REPORT=========================** Generic server 'service_agent-n1ql' terminating ** Last message in was {'DOWN',#Ref<0.3821750981.1062993923.31132>,process,                               <0.29327.6>,shutdown}** When Server state == {state,n1ql,                         {dict,2,16,16,8,80,48,                          {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},                          {{[],[],[],[],[],[],[],[],                            [[{node,'ns_1@172.23.136.103'}|                              <<"34f66682616ce6cd0dfe105dd712e77c">>]],                            [],[],[],[],[],                            [[{uuid,<<"34f66682616ce6cd0dfe105dd712e77c">>}|                              'ns_1@172.23.136.103']],                            []}}},                         undefined,undefined,undefined,undefined,undefined,                         undefined,undefined,                         {<<"AAAAAAAAAAI=">>,[]},                         {<<"AAAAAAAAAAI=">>,                          {topology,                           ['ns_1@172.23.136.103'],                           [<<"34f66682616ce6cd0dfe105dd712e77c">>],                           true,[]}},                         <0.29644.6>,<0.29645.6>,rebalance}** Reason for termination ==** {lost_connection,{'ns_1@172.23.136.103',shutdown}}
      [ns_server:debug,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:<0.30910.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.30795.6>} exited with reason normal[error_logger:error,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:service_agent-n1ql<0.28887.6>:ale_error_logger_handler:do_log:101]=========================CRASH REPORT=========================  crasher:    initial call: service_agent:init/1    pid: <0.28887.6>    registered_name: 'service_agent-n1ql'    exception exit: {lost_connection,{'ns_1@172.23.136.103',shutdown}}      in function  gen_server:handle_common_reply/8 (gen_server.erl, line 1241)    ancestors: [service_agent_children_sup,service_agent_sup,ns_server_sup,                  ns_server_nodes_sup,<0.27670.6>,ns_server_cluster_sup,                  root_sup,<0.152.0>]    message_queue_len: 2    messages: [{'EXIT',<0.29644.6>,                      {shutdown,                          {gen_server,call,                              [<0.29327.6>,                               {call,"ServiceAPI.GetTaskList",                                   #Fun<json_rpc_connection.0.36915653>,                                   #{timeout => 60000}},                               60000]}}},                  {'EXIT',<0.29645.6>,                      {shutdown,                          {gen_server,call,                              [<0.29327.6>,                               {call,"ServiceAPI.GetCurrentTopology",                                   #Fun<json_rpc_connection.0.36915653>,                                   #{timeout => 60000}},                               60000]}}}]    links: [<0.28889.6>,<0.28076.6>]    dictionary: []    trap_exit: true    status: running    heap_size: 6772    stack_size: 28    reductions: 29604  neighbours:
      [ns_server:debug,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:<0.28889.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.28887.6>} exited with reason {lost_connection,                                                                                  {'ns_1@172.23.136.103',                                                                                   shutdown}}[error_logger:error,2023-12-09T16:18:40.069-08:00,ns_1@172.23.136.103:service_agent_children_sup<0.28076.6>:ale_error_logger_handler:do_log:101]=========================SUPERVISOR REPORT=========================    supervisor: {local,service_agent_children_sup}    errorContext: child_terminated    reason: {lost_connection,{'ns_1@172.23.136.103',shutdown}}    offender: [{pid,<0.28887.6>},               {id,{service_agent,n1ql}},               {mfargs,{service_agent,start_link,[n1ql]}},               {restart_type,permanent},               {significant,false},               {shutdown,1000},               {child_type,worker}]
      [ns_server:debug,2023-12-09T16:18:40.070-08:00,ns_1@172.23.136.103:<0.30915.6>:ns_pubsub:do_subscribe_link_continue:150]Parent process of subscription {ns_config_events,<0.30805.6>} exited with reason normal[ns_server:debug,2023-12-09T16:18:40.070-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"goxdcr-cbauth",[{internal,true}],<0.28528.6>} needs_update[error_logger:info,2023-12-09T16:18:40.072-08:00,ns_1@172.23.136.103:service_agent_children_sup<0.28076.6>:ale_error_logger_handler:do_log:101]=========================PROGRESS REPORT=========================    supervisor: {local,service_agent_children_sup}    started: [{pid,<0.615.7>},              {id,{service_agent,n1ql}},              {mfargs,{service_agent,start_link,[n1ql]}},              {restart_type,permanent},              {significant,false},              {shutdown,1000},              {child_type,worker}]
      [ns_server:debug,2023-12-09T16:18:40.072-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"fts-cbauth",[{internal,true}],<0.28951.6>} needs_update[ns_server:debug,2023-12-09T16:18:40.080-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"index-cbauth",[{internal,true}],<0.28962.6>} needs_update[ns_server:debug,2023-12-09T16:18:40.085-08:00,ns_1@172.23.136.103:menelaus_cbauth<0.28063.6>:menelaus_cbauth:handle_cast:209]Observed json rpc process {"projector-cbauth",[{internal,true}],<0.29213.6>} needs_update[user:info,2023-12-09T16:18:40.103-08:00,ns_1@172.23.136.103:<0.27897.6>:ns_log:consume_log:76]Service 'n1ql' exited with status 2. Restarting. Messages:  C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:328 +0x1e3github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init(0xc0001983f0)  C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:166 +0x1a8github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).respawnProcess(0xc0001983f0)  C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:425 +0x118github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).receiveResponseMsg(0xc0001983f0)  C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:377 +0x232created by github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper.(*processEngine).Init in goroutine 932  C:/Jenkins/workspace/couchbase-server-windows/goproj/src/github.com/couchbase/eventing-ee/evaluator/impl/v8wrapper/process_engine.go:161 +0xc7
       


       

      Testrunner script to repro

      ./testrunner -i /tmp/testexec.3008.ini -p get-cbcollect-info=False,get-logs=False,get-coredumps=False,get-cbcollect-info=True,get-cbcollect-info=True -t clitest.docloadertest.docloaderTests.docloader_test,memory_quota=80,filename=beer-sample,standard_buckets=1,GROUP=P0

      Job name : windows11-os_certify-tools_1

      Job ref : https://cb-logs-qe.s3-website-us-west-2.amazonaws.com/7.6.0-1899/jenkins_logs/test_suite_executor-dynvm/18636/

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              raghav.sk Raghav S K
              raghav.sk Raghav S K
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty