Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-38283

memcached log wrapped due to excessive DCP "no memory" messages

    XMLWordPrintable

Details

    • Untriaged
    • Unknown

    Description

      As seen when investigating CBSE-8003, a large number of log messages are printed when DCP consumer stalls due to high memory usage:

      $ grep -c "Got error 'no memory' while trying to process mutation" memcached.log 
      207611
      

      However, the number of actual unique messages is small:

      $ grep "Got error 'no memory' while trying to process mutation" memcached.log | cut -d ' ' -f 2- | sort | uniq | wc -l
           108
      

      They are however intermixed, i.e. we see different vbids in consecutive log messages:

      $ grep "Got error 'no memory' while trying to process mutation" memcached.log |head -n 5
      2020-02-21T12:31:54.019542-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:309 Got error 'no memory' while trying to process mutation with seqno:153492
      2020-02-21T12:31:54.019545-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:311 Got error 'no memory' while trying to process mutation with seqno:163927
      2020-02-21T12:31:54.019548-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:883 Got error 'no memory' while trying to process mutation with seqno:159181
      2020-02-21T12:31:54.019552-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:894 Got error 'no memory' while trying to process mutation with seqno:164317
      2020-02-21T12:31:54.019555-07:00 WARNING 278: (bucket_1) DCP (Consumer) eq_dcpq:replication:ns_1@node_a->ns_1@node_b:bucket_1 - vb:890 Got error 'no memory' while trying to process mutation with seqno:160836
      

      We should modify the logging of this message to be edge-triggered instead of level-triggered. Would also be useful to add a log message when returning to a normal memory situation.

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-38283
          # Subject Branch Project Status CR V

          Activity

            drigby Dave Rigby created issue -
            drigby Dave Rigby made changes -
            Field Original Value New Value
            Link This issue relates to CBSE-8003 [ CBSE-8003 ]
            james.harrison James Harrison made changes -
            Assignee Daniel Owen [ owend ] James Harrison [ james.harrison ]
            wayne Wayne Siu made changes -
            Labels supportability 6.0.5-candidate supportability
            wayne Wayne Siu made changes -
            Fix Version/s 6.6.0 [ 16787 ]
            Fix Version/s 6.5.2 [ 16735 ]
            owend Daniel Owen made changes -
            Labels 6.0.5-candidate supportability supportability
            owend Daniel Owen made changes -
            Fix Version/s 6.0.5 [ 16729 ]
            drigby Dave Rigby made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            drigby Dave Rigby made changes -
            Labels supportability 6.0.5-candidate supportability
            owend Daniel Owen made changes -
            Fix Version/s 6.0.5 [ 16729 ]
            till Till Westmann made changes -
            Labels 6.0.5-candidate supportability 6.0.5-candidate approved-for-6.6.0 supportability
            till Till Westmann made changes -
            Link This issue blocks MB-38724 [ MB-38724 ]
            james.harrison James Harrison made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            owend Daniel Owen made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            owend Daniel Owen made changes -
            Assignee James Harrison [ james.harrison ] Daniel Owen [ owend ]
            drigby Dave Rigby made changes -
            Link This issue causes MB-41024 [ MB-41024 ]
            ritam.sharma Ritam Sharma made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Dave Rigby [ drigby ]
            wayne Wayne Siu made changes -
            Labels 6.0.5-candidate approved-for-6.6.0 supportability approved-for-6.0.5 approved-for-6.5.2 approved-for-6.6.0 supportability
            wayne Wayne Siu made changes -
            Fix Version/s 6.5.2 [ 17223 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-40511 [ MB-40511 ]
            wayne Wayne Siu made changes -
            Link This issue blocks MB-42583 [ MB-42583 ]
            owend Daniel Owen made changes -
            Assignee Dave Rigby [ drigby ] Paolo Cocchi [ paolo.cocchi ]
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Paolo Cocchi [ paolo.cocchi ] Daniel Owen [ owend ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            owend Daniel Owen made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            wayne Wayne Siu made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            owend Daniel Owen made changes -
            Assignee Daniel Owen [ owend ] Paolo Cocchi [ paolo.cocchi ]
            paolo.cocchi Paolo Cocchi made changes -
            Assignee Paolo Cocchi [ paolo.cocchi ] Ashwin Govindarajulu [ ashwin.govindarajulu ]
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            arunkumar Arunkumar Senthilnathan (Inactive) made changes -
            Labels approved-for-6.0.5 approved-for-6.5.2 approved-for-6.6.0 supportability approved-for-6.0.5 approved-for-6.5.2 approved-for-6.6.0 releasenote supportability
            ashwin.govindarajulu Ashwin Govindarajulu made changes -
            Status Resolved [ 5 ] Closed [ 6 ]

            People

              ashwin.govindarajulu Ashwin Govindarajulu
              drigby Dave Rigby
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty