Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-42506

[System Test] Mismatch between DCP buffer acks for Analytics DCP consumer

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-3625
      Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
      Scale : 3
      Iteration : 1st (Day 1)

      Seeing the following warnings on 172.23.121.77's memcached logs : 

      2020-11-04T14:25:37.442335-08:00 WARNING 152: (HISTORY) DCP (Producer) eq_dcpq:replication:ns_1@172.23.121.77->ns_1@172.23.97.74:HISTORY - (vb:428) Stream closing, sent until seqno 646 remaining items 0, reason: The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.507823-08:00 WARNING (HISTORY) DCP (Producer) eq_dcpq:cbas:Default.Local:HISTORY:cc40a04f742a3259f34cdd11aa083246:3 - Attempting to release 26345 bytes which is greater than bytesOutstanding:26342
      		 
      2020-11-04T14:25:37.508023-08:00 WARNING 487: exception occurred in runloop during packet execution. Closing connection: ThrowExceptionUnderflowPolicy current:26342 arg:26345. Cookies: [{"aiostat":"success","connection":"[ {\"ip\":\"172.23.120.74\",\"port\":43548} - {\"ip\":\"172.23.121.77\",\"port\":11210} (<ud>@cbas</ud>) ]","engine_storage":"0x00007f70200de100","ewouldblock":false,"packet":{"bodylen":4,"cas":0,"datatype":"raw","extlen":4,"key":"<ud></ud>","keylen":0,"magic":"ClientRequest","opaque":0,"opcode":"DCP_BUFFER_ACKNOWLEDGEMENT","vbucket":0},"refcount":1}]
      		 
      2020-11-04T14:25:37.508073-08:00 INFO 487: (No Engine) DCP (Producer) eq_dcpq:cbas:Default.Local:HISTORY:cc40a04f742a3259f34cdd11aa083246:3 - Removing connection [ {"ip":"172.23.120.74","port":43548} - {"ip":"172.23.121.77","port":11210} (<ud>@cbas</ud>) ]
      		 
      2020-11-04T14:25:37.508163-08:00 WARNING 487: (HISTORY) DCP (Producer) eq_dcpq:cbas:Default.Local:HISTORY:cc40a04f742a3259f34cdd11aa083246:3 - (vb:363) (sid:1) Stream closing, sent until seqno 516 remaining items 0, reason: The stream closed early because the conn was disconnected
      		 
      2020-11-04T14:25:37.508183-08:00 WARNING 487: (HISTORY) DCP (Producer) eq_dcpq:cbas:Default.Local:HISTORY:cc40a04f742a3259f34cdd11aa083246:3 - (vb:387) (sid:1) Stream closing, sent until seqno 489 remaining items 0, reason: The stream closed early because the conn was disconnected
      		 
      2020-11-04T14:25:37.508194-08:00 WARNING 487: (HISTORY) DCP (Producer) eq_dcpq:cbas:Default.Local:HISTORY:cc40a04f742a3259f34cdd11aa083246:3 - (vb:411) (sid:1) Stream closing, sent until seqno 666 remaining items 0, reason: The stream closed early because the conn was disconnected
      		 
      2020-11-04T14:25:37.532426-08:00 INFO 182: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.123.24->ns_1@172.23.121.77:HISTORY - (vb:897) Setting stream to dead state, last_seqno is 642, unAckedBytes is 0, status is The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.533736-08:00 INFO 182: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.123.24->ns_1@172.23.121.77:HISTORY - (vb:897) End stream received with reason The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.536794-08:00 INFO 165: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.121.77:HISTORY - (vb:974) Setting stream to dead state, last_seqno is 637, unAckedBytes is 0, status is The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.537587-08:00 INFO 165: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.97.74->ns_1@172.23.121.77:HISTORY - (vb:974) End stream received with reason The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.539042-08:00 INFO 163: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.123.33->ns_1@172.23.121.77:HISTORY - (vb:803) Setting stream to dead state, last_seqno is 524, unAckedBytes is 0, status is The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.540133-08:00 INFO 163: (HISTORY) DCP (Consumer) eq_dcpq:replication:ns_1@172.23.123.33->ns_1@172.23.121.77:HISTORY - (vb:803) End stream received with reason The stream closed early due to a close stream message
      		 
      2020-11-04T14:25:37.541789-08:00 INFO (HISTORY) VBucket::setState: transitioning vb:897 with high seqno:642 from:replica to:dead

      This is when a rebalance operation to add a new data node in the cluster is going on.

      This wasn't seen in the recent runs with 7.0.0 with the same system test.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MB-42591 The number of incoming DCP records is different from the target

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-42478 [CX] intermittent failure remote/cb/stats: remote-bucket-stats

          • Major - Major loss of function.
          • Closed
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MB-42588 [System Test] Mismatch between DCP buffer acks for FTS DCP consumer

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. MB-43317 Acknowledging too many bytes in DCP connection.

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-42588 [System Test] Mismatch between DCP buffer acks for FTS DCP consumer

          • Major - Major loss of function.
          • Resolved
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty