Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47318

DCP consumer acknowledgement under/over counting in replication flow control

    XMLWordPrintable

Details

    Description

      2021-07-11T07:28:58.094400+00:00 WARNING (default) DCP (Producer) eq_dcpq:replication:ns_1@172.23.107.102->ns_1@172.23.106.217:default - Attempting to release 7173586 bytes which is greater than bytesOutstanding:7173282
      

      Steps to Recreate:

      1. Create a 4 node cluster (RAM=256MB per node, replicas=1)
      2. Create 40 million items(doc size = 256)
      3. Start upserting all the existing 40 million items
      4. While upsert load is going on keep restarting couchbase server 5 times with a wait of 3s in between each restart
      5. Wait for the bucket to warmup.
      6. Repeat step 4 and 5 , Observed " Exception occurred during packet execution. Closing connection: ThrowExceptionUnderflowPolicy" in logs

      Memcached Logs:
      Error found on Node 172.23.107.102

      ERROR 45016: Exception occurred during packet execution. Closing connection: ThrowExceptionUnderflowPolicy current:7173282 arg:7173586. Cookies: [{"aiostat":"success","connection":"[ {\\"ip\\":\\"172.23.106.217\\",\\"port\\":55810} - {\\"ip\\":\\"172.23.107.102\\",\\"port\\":11209} (System, <ud>@ns_server</ud>) ]","engine_storage":"0x0000000000000000","ewouldblock":false,"packet":{"bodylen":4,"cas":0,"datatype":"raw","extlen":4,"key":"<ud></ud>","keylen":0,"magic":"ClientRequest","opaque":102,"opcode":"DCP_BUFFER_ACKNOWLEDGEMENT","vbucket":0},"refcount":1}] Exception thrown from: ["#0  /opt/couchbase/bin/memcached() [0x400000+0x9e454]","#1  /opt/couchbase/bin/memcached() [0x400000+0xc22fe]","#2  /opt/couchbase/bin/memcached() [0x400000+0x3c7cf4]","#3  /opt/couchbase/bin/memcached() [0x400000+0x3c7e4d]","#4  /opt/couchbase/bin/memcached() [0x400000+0x3c8054]","#5  /opt/couchbase/bin/memcached() [0x400000+0x288739]","#6  /opt/couchbase/bin/memcached() [0x400000+0x17ee4f]","#7  /opt/couchbase/bin/memcached() [0x400000+0x21b3b0]","#8  /opt/couchbase/bin/memcached() [0x400000+0x1e7aa2]","#9  /opt/couchbase/bin/memcached() [0x400000+0x1e94c0]","#10 /opt/couchbase/bin/memcached() [0x400000+0x1ce269]","#11 /opt/couchbase/bin/memcached() [0x400000+0x1ce623]","#12 /opt/couchbase/bin/memcached() [0x400000+0x1cec9b]","#13 /opt/couchbase/lib/libevent_core-2.1.so.7() [0x7fbbc63c5000+0xf8ae]","#14 /opt/couchbase/lib/libevent_core-2.1.so.7() [0x7fbbc63c5000+0x18821]","#15 /opt/couchbase/lib/libevent_core-2.1.so.7(event_base_loop+0x3bf) [0x7fbbc63c5000+0x1906f]","#16 /opt/couchbase/bin/memcached() [0x400000+0x766e47]","#17 /opt/couchbase/bin/memcached() [0x400000+0x767316]","#18 /opt/couchbase/bin/memcached() [0x400000+0x768cd6]","#19 /opt/couchbase/bin/memcached() [0x400000+0x1a0e29]","#20 /opt/couchbase/bin/memcached() [0x400000+0x6f97f9]","#21 /lib/x86_64-linux-gnu/libpthread.so.0() [0x7fbbc671c000+0x9609]","#22 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fbbc5ab2000+0x122293]"]\n']
      

      QE-TEST:

      guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.103712.ini bucket_storage=magma,rerun=false,GROUP=P0;256MB,randomize_value=true,bucket_eviction_policy=fullEviction,replicas=1,nodes_init=4,enable_dp=false,collect_pcaps=True,key_size=12,get-cbcollect-info=True,infra_log_level=critical,log_level=error,bucket_storage=magma,upgrade_version=7.1.0-1061 -t storage.magma.magma_crash_recovery.MagmaCrashTests.test_crash_during_ops,graceful=True,bucket_ram_quota=256,multiplier=1,batch_size=500,num_items=40000000,doc_size=256,doc_ops=update,GROUP=P0;256MB'
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-47318
          # Subject Branch Project Status CR V

          Activity

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              ankush.sharma Ankush Sharma
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are 3 open Gerrit changes

                  PagerDuty