Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61979

Eventing consumer process is continuously crashing with segmentation fault

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • 7.6.2
    • 7.6.2
    • eventing
    • Enterprise Edition 7.6.2 build 3658
    • Untriaged
    • Linux x86_64
    • 0
    • Yes

    Description

      ISSUE

      Eventing consumer process is continouosly crashing with segmentation fault after deploying single Eventing function called test.

      {"component":"eventing","uuid":"5f4ff686-678e-484f-b693-f7745255450d","timestamp":"2024-05-21T10:08:57.950Z","sub_component":"eventing-producer","severity":"info","event_id":4105,"description":"Import Functions","extra_attributes":{"appLocations":["test"]},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}
      {"component":"eventing","uuid":"2243959c-8cc5-4920-964b-a85e658c2931","timestamp":"2024-05-21T10:12:28.884Z","sub_component":"eventing-producer","severity":"info","event_id":4097,"description":"eventing-consumer process startup","extra_attributes":{"pid":13989,"workerName":"worker_test_0"},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}
      {"component":"eventing","uuid":"64ef427b-b68e-40e9-a556-d29f27509f6a","timestamp":"2024-05-21T10:12:35.140Z","sub_component":"eventing-producer","severity":"info","event_id":4109,"description":"Function deployed","extra_attributes":{"appLocation":"test"},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}
      {"component":"eventing","uuid":"aef9c57b-2505-4280-a89e-d652c6ad0f10","timestamp":"2024-05-21T10:13:11.018Z","sub_component":"eventing-producer","severity":"error","event_id":4098,"description":"eventing-consumer process crash","extra_attributes":{"pid":13989,"workerName":"worker_test_0"},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}
      {"component":"eventing","uuid":"b33726c4-4986-4eb3-808a-711fdf481ef0","timestamp":"2024-05-21T10:13:11.056Z","sub_component":"eventing-producer","severity":"info","event_id":4097,"description":"eventing-consumer process startup","extra_attributes":{"pid":14041,"workerName":"worker_test_0"},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}
      {"component":"eventing","uuid":"6e064c20-da2d-4a80-ab6d-7f8bd9ce3f4f","timestamp":"2024-05-21T10:13:12.376Z","sub_component":"eventing-producer","severity":"error","event_id":4098,"description":"eventing-consumer process crash","extra_attributes":{"pid":14041,"workerName":"worker_test_0"},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}
      {"component":"eventing","uuid":"7af3f4b4-3ef8-4135-85d9-ac8de7239e32","timestamp":"2024-05-21T10:13:12.413Z","sub_component":"eventing-producer","severity":"info","event_id":4097,"description":"eventing-consumer process startup","extra_attributes":{"pid":14055,"workerName":"worker_test_0"},"node":"172.23.106.67","otp_node":"ns_1@172.23.106.67"}

       

      Crash on Eventing node 172.23.106.67

      2024-05-21T03:43:31.778-07:00 [Info] eventing-consumer [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] == Minidump location: /opt/couchbase/var/lib/couchbase/crash/6eba4c4b-703d-48b0-bebd528e-c6a673ec.dmp Status: 1 ==
      2024-05-21T03:43:31.779-07:00 [Info] Consumer::sendUpdateProcessedSeqNo [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] vb: 793 seqNo: 0 sending update seqno data to C++
      2024-05-21T03:43:31.779-07:00 [Info] Consumer::processReqStreamMessages [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] vb: 845 reqStreamCh size: 178 msg: &consumer.streamRequestInfo{manifestUID:"0", startSeqNo:0x0, vb:0x34d, vbBlob:(*consumer.vbucketKVBlob)(0xc004f8d6c0)} Got request to stream
      2024-05-21T03:43:31.781-07:00 [Warn] client::Serve [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Failed to read from stderr pipe, err: EOF
      2024-05-21T03:43:31.781-07:00 [Error] Consumer::readMessageLoop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Read from client socket failed, err: read unix /tmp/127.0.0.1:8091_0_3433555026.sock->@: read: connection reset by peer
      2024-05-21T03:43:31.781-07:00 [Warn] client::Serve [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Failed to read from stdout pipe, err: EOF
      2024-05-21T03:43:31.781-07:00 [Error] Consumer::feedbackReadMessageLoop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Read from client socket failed, err: EOF
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::processReqStreamMessages [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] vb: 846 reqStreamCh size: 177 msg: &consumer.streamRequestInfo{manifestUID:"0", startSeqNo:0x0, vb:0x34e, vbBlob:(*consumer.vbucketKVBlob)(0xc004f8dba0)} Got request to stream
      2024-05-21T03:43:31.781-07:00 [Error] Consumer::sendMessage [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] stoppingConsumer: false write to downstream socket failed, err: write unix /tmp/127.0.0.1:8091_0_3433555026.sock->@: write: broken pipe
      2024-05-21T03:43:31.781-07:00 [Info] Producer::KillAndRespawnEventingConsumer [test1:0] IndexToPurge: 0 ConsumerIndex: 0 Shutting down Eventing.Consumer instance: consumer => app: test1 name: worker_test1_0 tcpPort: /tmp/127.0.0.1:8091_0_3433555026.sock ospid: 22760 dcpEventProcessed: DCP_STREAMREQ:795 DCP_SNAPSHOT:2 DCP_MUTATION:2 v8EventProcessed: thr_count:1 v8_init:1 execution_stats:1 log_level:1 v8_load:1 vb_map:1 failure_stats:1 latency_stats:1 lcb_exception_stats:1 curl_latency_stats:1 thr_map:1
      2024-05-21T03:43:31.781-07:00 [Info] Supervisor::Remove Token id: {1000727380214}
      2024-05-21T03:43:31.781-07:00 [Info] Producer::KillAndRespawnEventingConsumer [test1:0] IndexToPurge: 0 ConsumerIndex: 0 Closing down listener handles
      2024-05-21T03:43:31.781-07:00 [Error] Producer::handleV8Consumer [test1:0] Accept failed in main loop, err: accept unix /tmp/127.0.0.1:8091_0_3433555026.sock: use of closed network connection
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Gracefully shutting down consumer routine
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::RemoveSupervisorToken [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Removing supervisor token
      2024-05-21T03:43:31.781-07:00 [Info] Supervisor::Remove Token id: {2061584302080}
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Requested to remove supervision of eventing-consumer
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Stopped checkpoint, restart vb dcp stream and stats tickers
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::sendMessageLoop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Exiting send message routine
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Sent signal over channel to stop timer routines
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Sent signal to stop cpp worker stat collection routine
      2024-05-21T03:43:31.781-07:00 [Info] Consumer::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Sent signal over channel to stop checkpointing routine
      2024-05-21T03:43:31.781-07:00 [Info] client::Stop [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Exiting c++ worker
      2024-05-21T03:43:31.781-07:00 [Error] Producer::handleV8Consumer [test1:0] Accept failed in feedback loop, err: accept unix /tmp/f_127.0.0.1:8091_0_3433555026.sock: use of closed network connection
      2024-05-21T03:43:31.781-07:00 [Info] Producer::KillAndRespawnEventingConsumer [test1:0] ConsumerIndex: 0 respawning the Eventing.Consumer instance
      2024-05-21T03:43:31.781-07:00 [Info] Producer::handleV8Consumer [test1:0] udsSockPath len: 37 dump: /tmp/127.0.0.1:8091_0_3433555026.sock feedbackSockPath len: 39 dump: /tmp/f_127.0.0.1:8091_0_3433555026.sock
      2024-05-21T03:43:31.782-07:00 [Info] Producer::handleV8Consumer [test1:0] Spawning consumer to listen on socket: /tmp/127.0.0.1:8091_0_3433555026.sock feedback socket: /tmp/f_127.0.0.1:8091_0_3433555026.sock index: 0 vbs len: 1024 dump: [0-1023]
      2024-05-21T03:43:31.783-07:00 [Warn] client::Serve [worker_test1_0:/tmp/127.0.0.1:8091_0_3433555026.sock:22760] Exiting c++ worker with error: signal: segmentation fault
      

      Attachments

        1. bt_full.txt
          7 kB
        2. info_thread.txt
          2 kB
        3. threads_apply_all_bt.txt
          13 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sujay.gad Sujay Gad
              sujay.gad Sujay Gad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty