Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.4, 6.5.1
-
Untriaged
-
Unknown
Description
As seen when investigating CBSE-8003, a large number of log messages are printed when DCP consumer stalls due to high memory usage:
$ grep -c "Got error 'no memory' while trying to process mutation" memcached.log
|
207611
|
However, the number of actual unique messages is small:
$ grep "Got error 'no memory' while trying to process mutation" memcached.log | cut -d ' ' -f 2- | sort | uniq | wc -l
|
108
|
They are however intermixed, i.e. we see different vbids in consecutive log messages:
$ grep "Got error 'no memory' while trying to process mutation" memcached.log |head -n 5
|
2020-02-21T12:31:54.019542-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:309 Got error 'no memory' while trying to process mutation with seqno:153492
|
2020-02-21T12:31:54.019545-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:311 Got error 'no memory' while trying to process mutation with seqno:163927
|
2020-02-21T12:31:54.019548-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:883 Got error 'no memory' while trying to process mutation with seqno:159181
|
2020-02-21T12:31:54.019552-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:894 Got error 'no memory' while trying to process mutation with seqno:164317
|
2020-02-21T12:31:54.019555-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:890 Got error 'no memory' while trying to process mutation with seqno:160836
|
We should modify the logging of this message to be edge-triggered instead of level-triggered. Would also be useful to add a log message when returning to a normal memory situation.
Attachments
Issue Links
- causes
-
-
- Closed
-