Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-29840

TMP OOM failures during swap rebalance with takeover_via_orchestrator

    XMLWordPrintable

Details

    Description

      Build 5.1.1-5721.

      4 nodes, swap rebalance (172.23.96.103 -> 172.23.96.104), 1B items, 15K ops/sec (90/10 R/W), 10%% cache miss rate.

      [6:09 PM] I think it might be cursor dropping - maybe
      [6:12 PM] all the tmp ooms occur on .104 between when the memory is above the HWM
      [6:13 PM] Can you create a ticket and upload the logs from this run and the logs from the previous run?

      Logs from 5.1.1-5720: (no quirks)
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-264/172.23.96.100.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-264/172.23.96.101.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-264/172.23.96.102.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-264/172.23.96.103.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-264/172.23.96.104.zip

      Logs from 5.1.1-5715: (no quirks)
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-229/172.23.96.100.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-229/172.23.96.101.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-229/172.23.96.102.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-229/172.23.96.103.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-229/172.23.96.104.zip

      Logs from 5.1.1-5710: (no quirks)
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-208/172.23.96.100.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-208/172.23.96.101.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-208/172.23.96.102.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-208/172.23.96.103.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-208/172.23.96.104.zip

      Logs from 5.1.0-5552:
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-142/172.23.96.100.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-142/172.23.96.101.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-142/172.23.96.102.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-142/172.23.96.103.zip
      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-titan-reb-142/172.23.96.104.zip

      Attachments

        1. gets - 5720.png
          gets - 5720.png
          149 kB
        2. gets and ooms.png
          gets and ooms.png
          184 kB
        3. mem_used.png
          mem_used.png
          339 kB
        4. mem_used and HWM.png
          mem_used and HWM.png
          186 kB
        5. mem_used and HWM - 5720.png
          mem_used and HWM - 5720.png
          188 kB
        6. mem_used in 5715.png
          mem_used in 5715.png
          348 kB
        7. mem_used in 5720.png
          mem_used in 5720.png
          340 kB
        8. tmp-ooms.png
          tmp-ooms.png
          188 kB
        9. vb_num_5720.png
          vb_num_5720.png
          189 kB
        10. vb_num_5721.png
          vb_num_5721.png
          187 kB

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-26738 memcached fails to serve KV requests during swap rebalance due to OOM failures

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              owend Daniel Owen
              pavelpaulau Pavel Paulau (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty