Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-27772

Missed events when destination bucket is 2% DGM, doc size 2MB

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • 5.5.0
    • 5.5.0
    • eventing
    • Untriaged
    • Centos 64-bit
    • No

    Description

      Script to Repro:

      ./testrunner -i /tmp/win10-bucket-ops.ini  -t eventing.eventing_dataset.EventingDataset.test_eventing_with_large_doc_size,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True,document_size=2048000
      

      The script basically pushes the src and dst buckets to less than 2% resident ratio. After creating 2016 docs of 2MB each, we deploy the following handler code which copies data from src bucket to dst bucket.

      function OnUpdate(doc, meta) {
          var doc_id = meta.id;
          log('creating document for : ', doc);
          dst_bucket[doc_id] = doc; // SET operation
      }
      function OnDelete(meta) {
          log('deleting document', meta.id);
          delete dst_bucket[meta.id]; // DELETE operation
      } 
      

      In this test only 1,573 docs got transferred to dst_bucket.

      Some important logs.

      [root@s44009 ~]# dmesg | egrep -i 'eventing'
      eventing-consum invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
      eventing-consum cpuset=/ mems_allowed=0
      Pid: 9393, comm: eventing-consum Not tainted 2.6.32-573.7.1.el6.x86_64 #1
      [ 8735]   498  8735   156152    41220   1       0             0 eventing-produc
      [ 9374]   498  9374   523705   271048   0       0             0 eventing-consum
      [ 9376]   498  9376   509769   257668   2       0             0 eventing-consum
      [ 9379]   498  9379   521289   253174   3       0             0 eventing-consum
      Out of memory: Kill process 9374 (eventing-consum) score 135 or sacrifice child
      Killed process 9374, UID 498, (eventing-consum) total-vm:2094820kB, anon-rss:1081132kB, file-rss:3060kB
      [root@s44009 ~]# ps -ef | grep eventing
      498       8735  8730 37 22:27 ?        00:10:53 /opt/couchbase/bin/eventing-producer -adminport=8096 -dir=/opt/couchbase/var/lib/couchbase/data/@eventing -kvport=11210 -restport=8091 -uuid=83c2315153c3794f89f036e7e80a7484 -diagdir=/opt/couchbase/var/lib/couchbase/crash -ipv6=false -vbuckets=1024 -adminsslport=18096 -certfile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem -keyfile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem
      498       9376  8735 13 22:39 ?        00:02:21 eventing-consumer Function_219673367_test_eventing_with_large_doc_size af_unix /tmp/127.0.0.1:8091_worker_Function_219673367_test_eventing_with_large_doc_size_1.sock worker_Function_219673367_test_eventing_with_large_doc_size_1 1 /opt/couchbase/var/lib/couchbase/crash 8096
      498       9379  8735 14 22:39 ?        00:02:23 eventing-consumer Function_219673367_test_eventing_with_large_doc_size af_unix /tmp/127.0.0.1:8091_worker_Function_219673367_test_eventing_with_large_doc_size_2.sock worker_Function_219673367_test_eventing_with_large_doc_size_2 1 /opt/couchbase/var/lib/couchbase/crash 8096
      498       9605  8735  0 22:43 ?        00:00:00 eventing-consumer Function_219673367_test_eventing_with_large_doc_size af_unix /tmp/127.0.0.1:8091_worker_Function_219673367_test_eventing_with_large_doc_size_0.sock worker_Function_219673367_test_eventing_with_large_doc_size_0 1 /opt/couchbase/var/lib/couchbase/crash 8096
      root     10401 10328  0 22:56 pts/0    00:00:00 grep eventing
      [root@s44009 ~]# cd /opt/couchbase/var/lib/couchbase/crash
      [root@s44009 crash]# ls -lrt
      total 0
      [root@s44009 crash]# 
      

      Logs attached.

      Attachments

        1. eventing.zip
          183 kB
        2. test_16.zip
          40.38 MB
        3. test_17.zip
          42.73 MB
        4. test_23.zip
          57.60 MB
        5. test_24.zip
          60.79 MB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Gautham.Banasandra Gautham Banasandra (Inactive)
              Balakumaran.Gopal Balakumaran Gopal
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty