Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61978

[System Test] Analytics service unavailable - JVM halting with status 88 errors seen (S3 rate limiting)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • Columnar 1.0.0
    • Columnar 1.0.0
    • analytics
    • Columnar Edition 1.0.0 build 2085 4-node cluster ( 8vCPUs, 32GB instance type)
    • Untriaged
    • 0
    • Unknown
    • Analytics Sprint 43, Analytics Sprint 44

    Description

      During the system test run (day 2 into the run), it was seen that creation of remote links failed with an internal server error - 

       {"env": "other", "method": "POST", "url": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/61224aa7-468c-455c-a645-a8f599579b52/projects/c5accdec-d04a-4b29-8df1-3be82ca6fc22/instance/17e7a7c4-a39f-4d50-b865-103b7142675c/proxy/query/service", "timeoutSec": 600, "attempt": 1, "method": "POST", "address": "https://api.qe-10.sandbox.nonprod-project-avengers.com/v2/organizations/61224aa7-468c-455c-a645-a8f599579b52/projects/c5accdec-d04a-4b29-8df1-3be82ca6fc22/instance/17e7a7c4-a39f-4d50-b865-103b7142675c/proxy/query/service", "error": "unexpected status code 500", "status_code": 500, "requestId": "1334d06c-1bcd-4db0-8b66-c48c0148d7ad"}

      The logs seem to suggest exceptions -

      on node 004-

       b'2024-05-21T10:27:19.334+00:00 FATA CBAS.util.ExitUtil [Executor-4505:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-4505:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:27:19.944+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:28:35.222+00:00 FATA CBAS.util.ExitUtil [Executor-308:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-308:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:28:35.596+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:29:37.255+00:00 FATA CBAS.util.ExitUtil [Executor-235:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-235:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:29:37.649+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:30:26.274+00:00 FATA CBAS.util.ExitUtil [Executor-239:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-239:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:30:26.686+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:31:36.709+00:00 FATA CBAS.util.ExitUtil [Executor-270:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-270:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:31:37.070+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:32:54.968+00:00 FATA CBAS.util.ExitUtil [Executor-253:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-253:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:32:55.293+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:34:06.396+00:00 FATA CBAS.util.ExitUtil [Executor-211:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-211:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:34:06.653+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:27:19.334+00:00 FATA CBAS.util.ExitUtil [Executor-4505:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-4505:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:27:19.944+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:28:35.222+00:00 FATA CBAS.util.ExitUtil [Executor-308:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-308:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:28:35.596+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:29:37.255+00:00 FATA CBAS.util.ExitUtil [Executor-235:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-235:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:29:37.649+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:30:26.274+00:00 FATA CBAS.util.ExitUtil [Executor-239:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-239:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:30:26.686+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:31:36.709+00:00 FATA CBAS.util.ExitUtil [Executor-270:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-270:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:31:37.070+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:32:54.968+00:00 FATA CBAS.util.ExitUtil [Executor-253:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-253:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:32:55.293+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:34:06.396+00:00 FATA CBAS.util.ExitUtil [Executor-211:6da35fd8abcdef381c5ff937b6e5551a] JVM halting with status 88 (halting thread Thread[Executor-211:6da35fd8abcdef381c5ff937b6e5551a,5,main], interrupted false)\n2024-05-21T10:34:06.653+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \norg.apache.hyracks.api.exceptions.HyracksDataException: ASX1108: External source error. java.net.SocketException: Connection

      Similar messages are seen on the other nodes as well. For eg. on node 002

       b'2024-05-21T10:27:27.031+00:00 FATA CBAS.util.ExitUtil [Executor-6231:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-6231:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:27:27.865+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:28:27.318+00:00 FATA CBAS.util.ExitUtil [Executor-239:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-239:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:28:27.560+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:29:38.317+00:00 FATA CBAS.util.ExitUtil [Executor-220:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-220:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:29:38.610+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:30:36.797+00:00 FATA CBAS.util.ExitUtil [Executor-239:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-239:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:30:37.120+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:31:39.615+00:00 FATA CBAS.util.ExitUtil [Executor-236:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-236:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:31:39.887+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:33:01.132+00:00 FATA CBAS.util.ExitUtil [Executor-336:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-336:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:33:01.505+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:34:15.235+00:00 FATA CBAS.util.ExitUtil [Executor-222:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-222:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:34:15.514+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:27:27.031+00:00 FATA CBAS.util.ExitUtil [Executor-6231:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-6231:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:27:27.865+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:28:27.318+00:00 FATA CBAS.util.ExitUtil [Executor-239:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-239:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:28:27.560+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:29:38.317+00:00 FATA CBAS.util.ExitUtil [Executor-220:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-220:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:29:38.610+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:30:36.797+00:00 FATA CBAS.util.ExitUtil [Executor-239:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-239:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:30:37.120+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:31:39.615+00:00 FATA CBAS.util.ExitUtil [Executor-236:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-236:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:31:39.887+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt: \n2024-05-21T10:33:01.132+00:00 FATA CBAS.util.ExitUtil [Executor-336:fee19ca04704c8d360afcc9c27a735d6] JVM halting with status 88 (halting thread Thread[Executor-336:fee19ca04704c8d360afcc9c27a735d6,5,main], interrupted false)\n2024-05-21T10:33:01.505+00:00 FATA CBAS.util.ExitUtil [pool-2-thread-1] Thread dump at halt:

      cbcollect ->

      https://cb-engineering.s3.amazonaws.com/SysTestColumnar20May/collectinfo-2024-05-21T103153-ns_1%40svc-da-node-001.f7x3yra8xmrvzlw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar20May/collectinfo-2024-05-21T103153-ns_1%40svc-da-node-002.f7x3yra8xmrvzlw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar20May/collectinfo-2024-05-21T103153-ns_1%40svc-da-node-003.f7x3yra8xmrvzlw.sandbox.nonprod-project-avengers.com.zip
      https://cb-engineering.s3.amazonaws.com/SysTestColumnar20May/collectinfo-2024-05-21T103153-ns_1%40svc-da-node-004.f7x3yra8xmrvzlw.sandbox.nonprod-project-avengers.com.zip

      Attachments

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              PagerDuty