Details
-
Bug
-
Resolution: Fixed
-
Critical
-
6.0.4, 6.5.1
-
Untriaged
-
Unknown
Description
As seen when investigating CBSE-8003, a large number of log messages are printed when DCP consumer stalls due to high memory usage:
$ grep -c "Got error 'no memory' while trying to process mutation" memcached.log
|
207611
|
However, the number of actual unique messages is small:
$ grep "Got error 'no memory' while trying to process mutation" memcached.log | cut -d ' ' -f 2- | sort | uniq | wc -l
|
108
|
They are however intermixed, i.e. we see different vbids in consecutive log messages:
$ grep "Got error 'no memory' while trying to process mutation" memcached.log |head -n 5
|
2020-02-21T12:31:54.019542-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:309 Got error 'no memory' while trying to process mutation with seqno:153492
|
2020-02-21T12:31:54.019545-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:311 Got error 'no memory' while trying to process mutation with seqno:163927
|
2020-02-21T12:31:54.019548-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:883 Got error 'no memory' while trying to process mutation with seqno:159181
|
2020-02-21T12:31:54.019552-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:894 Got error 'no memory' while trying to process mutation with seqno:164317
|
2020-02-21T12:31:54.019555-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:890 Got error 'no memory' while trying to process mutation with seqno:160836
|
We should modify the logging of this message to be edge-triggered instead of level-triggered. Would also be useful to add a log message when returning to a normal memory situation.
Attachments
Issue Links
- causes
-
MB-41024 Crash in DCP consumer when processing SyncWrite Prepare at end of snapshot under memory pressure
- Closed
For Gerrit Dashboard: MB-38283 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
124383,3 | MB-38283: Stop PassiveStream repeatedly logging ENOMEM | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
127403,1 | Merge branch 'couchbase/mad-hatter' into 'couchbase/master' | master | kv_engine | Status: MERGED | +2 | +1 |
134816,4 | MB-41024: PassiveStream: Don't handleSnapshotEnd if message not processed | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
134830,1 | MB-41024: PassiveStream: Don't handleSnapshotEnd if message not processed | master | kv_engine | Status: ABANDONED | 0 | -1 |
142294,1 | MB-38283: Stop PassiveStream repeatedly logging ENOMEM | alice | kv_engine | Status: ABANDONED | 0 | 0 |
142428,2 | MB-38283: Stop PassiveStream repeatedly logging ENOMEM | alice | kv_engine | Status: MERGED | +2 | +1 |
142824,2 | MB-38283: [BP] Stop PassiveStream repeatedly logging ENOMEM | 6.5.2 | kv_engine | Status: MERGED | +2 | +1 |
142979,3 | Merge remote-tracking branch 'couchbase/6.5.2' into mad-hatter | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
143693,1 | Merge branch 'mad-hatter' | master | kv_engine | Status: MERGED | +2 | +1 |
143885,1 | Merge branch 'alice' into mad-hatter | mad-hatter | kv_engine | Status: MERGED | +2 | +1 |
144505,1 | Merge mad-hatter into master | master | kv_engine | Status: MERGED | +2 | +1 |