Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38283

memcached log wrapped due to excessive DCP "no memory" messages

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      As seen when investigating CBSE-8003, a large number of log messages are printed when DCP consumer stalls due to high memory usage:

      $ grep -c "Got error 'no memory' while trying to process mutation" memcached.log 
      207611
      

      However, the number of actual unique messages is small:

      $ grep "Got error 'no memory' while trying to process mutation" memcached.log | cut -d ' ' -f 2- | sort | uniq | wc -l
           108
      

      They are however intermixed, i.e. we see different vbids in consecutive log messages:

      $ grep "Got error 'no memory' while trying to process mutation" memcached.log |head -n 5
      2020-02-21T12:31:54.019542-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:309 Got error 'no memory' while trying to process mutation with seqno:153492
      2020-02-21T12:31:54.019545-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:311 Got error 'no memory' while trying to process mutation with seqno:163927
      2020-02-21T12:31:54.019548-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:883 Got error 'no memory' while trying to process mutation with seqno:159181
      2020-02-21T12:31:54.019552-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:894 Got error 'no memory' while trying to process mutation with seqno:164317
      2020-02-21T12:31:54.019555-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:890 Got error 'no memory' while trying to process mutation with seqno:160836
      

      We should modify the logging of this message to be edge-triggered instead of level-triggered. Would also be useful to add a log message when returning to a normal memory situation.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-38283
          # Subject Branch Project Status CR V

          Activity

            drigby Dave Rigby created issue -
            drigby Dave Rigby made changes -
            Field Original Value New Value
            Link This issue relates to CBSE-8003 [ CBSE-8003 ]
            james.harrison James Harrison made changes -
            Assignee Daniel Owen [ owend ] James Harrison [ james.harrison ]
            wayne Wayne Siu made changes -
            Labels supportability 6.0.5-candidate supportability
            wayne Wayne Siu made changes -
            Fix Version/s 6.6.0 [ 16787 ]
            Fix Version/s 6.5.2 [ 16735 ]
            owend Daniel Owen made changes -
            Labels 6.0.5-candidate supportability supportability
            owend Daniel Owen made changes -
            Fix Version/s 6.0.5 [ 16729 ]
            drigby Dave Rigby made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            drigby Dave Rigby made changes -
            Labels supportability 6.0.5-candidate supportability
            owend Daniel Owen made changes -
            Fix Version/s 6.0.5 [ 16729 ]
            till Till Westmann made changes -
            Labels 6.0.5-candidate supportability 6.0.5-candidate approved-for-6.6.0 supportability
            till Till Westmann made changes -
            Link This issue blocks MB-38724 [ MB-38724 ]

            Build couchbase-server-6.6.0-7565 contains kv_engine commit d13d1ff with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7565 contains kv_engine commit d13d1ff with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM

            Build couchbase-server-7.0.0-1999 contains kv_engine commit d13d1ff with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-1999 contains kv_engine commit d13d1ff with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM
            james.harrison James Harrison made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            owend Daniel Owen made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            owend Daniel Owen made changes -
            Assignee James Harrison [ james.harrison ] Daniel Owen [ owend ]
            drigby Dave Rigby made changes -
            Link This issue causes MB-41024 [ MB-41024 ]
            ritam.sharma Ritam Sharma added a comment -

            Daniel Owen - Can you please check if this change is back ported for 6.0.5.

            ritam.sharma Ritam Sharma added a comment - Daniel Owen - Can you please check if this change is back ported for 6.0.5.
            ritam.sharma Ritam Sharma made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Dave Rigby [ drigby ]
            owend Daniel Owen added a comment -

            Hi Dave Rigby Do you mind taking a look at this - does not look like the BP to 6.0.5 was done?

            Not sure its worth it..

            thanks

            owend Daniel Owen added a comment - Hi Dave Rigby Do you mind taking a look at this - does not look like the BP to 6.0.5 was done? Not sure its worth it.. thanks
            wayne Wayne Siu made changes -
            Labels 6.0.5-candidate approved-for-6.6.0 supportability approved-for-6.0.5 approved-for-6.5.2 approved-for-6.6.0 supportability
            wayne Wayne Siu made changes -
            Fix Version/s 6.5.2 [ 17223 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-40511 [ MB-40511 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-42583 [ MB-42583 ]
            owend Daniel Owen made changes -
            Assignee Dave Rigby [ drigby ] Paolo Cocchi [ paolo.cocchi ]
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Paolo Cocchi [ paolo.cocchi ] Daniel Owen [ owend ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

            Build couchbase-server-6.0.4-3111 contains kv_engine commit b8aa082 with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.4-3111 contains kv_engine commit b8aa082 with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM

            Build couchbase-server-6.0.5-3335 contains kv_engine commit b8aa082 with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.0.5-3335 contains kv_engine commit b8aa082 with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM
            owend Daniel Owen made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            wayne Wayne Siu added a comment -

            Paolo Cocchi Daniel Owen

            Have you also merged these changes to 6.5.2?  Thanks.

            wayne Wayne Siu added a comment - Paolo Cocchi Daniel Owen Have you also merged these changes to 6.5.2?  Thanks.
            wayne Wayne Siu made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Paolo Cocchi [ paolo.cocchi ]

            Validated the fix on 6.0.5-3338.

            Steps:

            1. 3 node cluster with cb bucket (replica=1)
            2. Loaded docs into cbc-pillowfight and pushed the DGM to very low values
            3. Rebalanced out node-3 while pillowfight still running

            Seeing both the commands giving same count,

            grep -c "Got error 'no memory' while trying to process mutation" memcached.*
            grep "Got error 'no memory' while trying to process mutation" memcached* | cut -d ' ' -f 2- | sort | uniq | wc -l
            

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated the fix on 6.0.5-3338. Steps: 3 node cluster with cb bucket (replica=1) Loaded docs into cbc-pillowfight and pushed the DGM to very low values Rebalanced out node-3 while pillowfight still running Seeing both the commands giving same count, grep -c "Got error 'no memory' while trying to process mutation" memcached.* grep "Got error 'no memory' while trying to process mutation" memcached* | cut -d ' ' -f 2- | sort | uniq | wc -l
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Paolo Cocchi [ paolo.cocchi ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

            Build couchbase-server-6.5.2-6607 contains kv_engine commit ba93a67 with commit message:
            MB-38283: [BP] Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.2-6607 contains kv_engine commit ba93a67 with commit message: MB-38283 : [BP] Stop PassiveStream repeatedly logging ENOMEM

            Build couchbase-server-6.6.2-9417 contains kv_engine commit ba93a67 with commit message:
            MB-38283: [BP] Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9417 contains kv_engine commit ba93a67 with commit message: MB-38283 : [BP] Stop PassiveStream repeatedly logging ENOMEM
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Labels approved-for-6.0.5 approved-for-6.5.2 approved-for-6.6.0 supportability approved-for-6.0.5 approved-for-6.5.2 approved-for-6.6.0 releasenote supportability

            Validated the fix on 6.5.2-6607.

            Closing this ticket.

            ashwin.govindarajulu Ashwin Govindarajulu added a comment - Validated the fix on 6.5.2-6607. Closing this ticket.
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            Build couchbase-server-7.0.0-4255 contains kv_engine commit ba93a67 with commit message:
            MB-38283: [BP] Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4255 contains kv_engine commit ba93a67 with commit message: MB-38283 : [BP] Stop PassiveStream repeatedly logging ENOMEM

            Build couchbase-server-6.6.2-9447 contains kv_engine commit b8aa082 with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9447 contains kv_engine commit b8aa082 with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM

            Build couchbase-server-7.0.0-4322 contains kv_engine commit b8aa082 with commit message:
            MB-38283: Stop PassiveStream repeatedly logging ENOMEM

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4322 contains kv_engine commit b8aa082 with commit message: MB-38283 : Stop PassiveStream repeatedly logging ENOMEM

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty