Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38283

memcached log wrapped due to excessive DCP "no memory" messages

    XMLWordPrintable

    Details

    • Triage:
      Untriaged
    • Is this a Regression?:
      Unknown

      Description

      As seen when investigating CBSE-8003, a large number of log messages are printed when DCP consumer stalls due to high memory usage:

      $ grep -c "Got error 'no memory' while trying to process mutation" memcached.log 
      207611
      

      However, the number of actual unique messages is small:

      $ grep "Got error 'no memory' while trying to process mutation" memcached.log | cut -d ' ' -f 2- | sort | uniq | wc -l
           108
      

      They are however intermixed, i.e. we see different vbids in consecutive log messages:

      $ grep "Got error 'no memory' while trying to process mutation" memcached.log |head -n 5
      2020-02-21T12:31:54.019542-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:309 Got error 'no memory' while trying to process mutation with seqno:153492
      2020-02-21T12:31:54.019545-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:311 Got error 'no memory' while trying to process mutation with seqno:163927
      2020-02-21T12:31:54.019548-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:883 Got error 'no memory' while trying to process mutation with seqno:159181
      2020-02-21T12:31:54.019552-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:894 Got error 'no memory' while trying to process mutation with seqno:164317
      2020-02-21T12:31:54.019555-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:890 Got error 'no memory' while trying to process mutation with seqno:160836
      

      We should modify the logging of this message to be edge-triggered instead of level-triggered. Would also be useful to add a log message when returning to a normal memory situation.

        Attachments

          Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

            Activity

            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9417 contains kv_engine commit ba93a67 with commit message:
            MB-38283: [BP] Stop PassiveStream repeatedly logging ENOMEM

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9417 contains kv_engine commit ba93a67 with commit message: MB-38283 : [BP] Stop PassiveStream repeatedly logging ENOMEM
            Hide
            ashwin.govindarajulu Ashwin Govindarajulu added a comment -

            Validated the fix on 6.5.2-6607.

            Closing this ticket.

            Show
            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated the fix on 6.5.2-6607. Closing this ticket.
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-4255 contains kv_engine commit ba93a67 with commit message:
            MB-38283: [BP] Stop PassiveStream repeatedly logging ENOMEM

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4255 contains kv_engine commit ba93a67 with commit message: MB-38283 : [BP] Stop PassiveStream repeatedly logging ENOMEM
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-6.6.2-9447 contains kv_engine commit b8aa082 with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9447 contains kv_engine commit b8aa082 with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM
            Hide
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.0-4322 contains kv_engine commit b8aa082 with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            Show
            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4322 contains kv_engine commit b8aa082 with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM

              People

              Assignee:
              ashwin.govindarajulu Ashwin Govindarajulu
              Reporter:
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  PagerDuty