Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-40543

Add the ability to re-enable FTS logging in memcached for debug purposes

    XMLWordPrintable

    Details

    • Story Points:
      1

      Description

      In MB-28468 (5.5.x) the log level for FTS connections to memcached was reduced, to prevent the overly-noisy logging from wrapping the memcached logs too soon.

      This is fixed with improvements in CC/7.0, and the logging has been re-enabled in memcached.

      In the meantime however (6.5/6.6), we have no way of debugging the DCP streams for problematic FTS index builds.

      Please add a mechanism to allow the re-enabling of logs from FTS for this purpose - default should be the same behaviour.

        Attachments

        For Gerrit Dashboard: MB-40543
        # Subject Branch Project Status CR V

          Activity

          Hide
          mihir.kamdar Mihir Kamdar added a comment -

          Daniel Owen is there going to be some setting/configuration that will allow re-enabling of logs from FTS ?

          Show
          mihir.kamdar Mihir Kamdar added a comment - Daniel Owen is there going to be some setting/configuration that will allow re-enabling of logs from FTS ?
          Hide
          ben.huddleston Ben Huddleston added a comment -

          Could do one of two things here.

          1. Create a config variable to disable the blacklist of FTS connection logging. This would treat FTS connections in the same way as any other.
          2. Create a config variable to set the FTS connection log level explicitly. This may make debugging easier as the FTS connections could be set to a debug level without setting the reset of the memcached process to a debug level.

          They're both about the same amount of work.

          Show
          ben.huddleston Ben Huddleston added a comment - Could do one of two things here. Create a config variable to disable the blacklist of FTS connection logging. This would treat FTS connections in the same way as any other. Create a config variable to set the FTS connection log level explicitly. This may make debugging easier as the FTS connections could be set to a debug level without setting the reset of the memcached process to a debug level. They're both about the same amount of work.
          Hide
          drigby Dave Rigby added a comment - - edited

          Fix merged to kv_engine/6.5.1 branch (for 6.5.1-MP2).

          Add a new config option:

          • dcp_blacklist_fts_connection_logs - Blacklists FTS DCP logging by
            default by setting the log level to critical and unregistering the
            logger from log level verbosity changes. Defaults to true.

          To un-blacklist FTS, change this to false.

          Usage:

          Set either via bucket config, or epctl on a per node / per bucket basis:

          cbepctl <HOST> -u Administrator -p asdasd -b <BUCKET> set dcp_param dcp_blacklist_fts_connection_logs false

          Show
          drigby Dave Rigby added a comment - - edited Fix merged to kv_engine/6.5.1 branch (for 6.5.1-MP2). Add a new config option: dcp_blacklist_fts_connection_logs - Blacklists FTS DCP logging by default by setting the log level to critical and unregistering the logger from log level verbosity changes. Defaults to true. To un-blacklist FTS, change this to false. Usage: Set either via bucket config, or epctl on a per node / per bucket basis: cbepctl <HOST> -u Administrator -p asdasd -b <BUCKET> set dcp_param dcp_blacklist_fts_connection_logs false
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.5.1-6302 contains kv_engine commit 166a75a with commit message:
          MB-40543: Add dynamic blacklist FTS log config option

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.5.1-6302 contains kv_engine commit 166a75a with commit message: MB-40543 : Add dynamic blacklist FTS log config option
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.0-7897 contains kv_engine commit 8cec59e with commit message:
          MB-40543: Merge branch '6.5.1' into mad-hatter

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7897 contains kv_engine commit 8cec59e with commit message: MB-40543 : Merge branch '6.5.1' into mad-hatter
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-6.6.0-7897 contains kv_engine commit 166a75a with commit message:
          MB-40543: Add dynamic blacklist FTS log config option

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.0-7897 contains kv_engine commit 166a75a with commit message: MB-40543 : Add dynamic blacklist FTS log config option
          Hide
          mihir.kamdar Mihir Kamdar added a comment -

          Validated on 6.5.1-6302 and 6.6.0-7897.

          memcached logs have logs entries like the below ones after un-blacklisting FTS connection logs

          memcached.log.000004.txt:2020-07-24T14:29:31.029040-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:206) ActiveStream::transitionState: Transitioning from backfilling to in-memory
          memcached.log.000004.txt:2020-07-24T14:29:31.431235-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:226) Backfill complete, 11357 items read from disk, 638 from memory, last seqno read: 47582, pendingBackfill : False
          memcached.log.000004.txt:2020-07-24T14:29:31.456859-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:226) ActiveStream::transitionState: Transitioning from backfilling to in-memory
          memcached.log.000004.txt:2020-07-24T14:29:32.062066-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:216) Backfill complete, 11326 items read from disk, 643 from memory, last seqno read: 95680, pendingBackfill : False
          memcached.log.000004.txt:2020-07-24T14:29
          

          Show
          mihir.kamdar Mihir Kamdar added a comment - Validated on 6.5.1-6302 and 6.6.0-7897. memcached logs have logs entries like the below ones after un-blacklisting FTS connection logs memcached.log.000004.txt:2020-07-24T14:29:31.029040-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:206) ActiveStream::transitionState: Transitioning from backfilling to in-memory memcached.log.000004.txt:2020-07-24T14:29:31.431235-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:226) Backfill complete, 11357 items read from disk, 638 from memory, last seqno read: 47582, pendingBackfill : False memcached.log.000004.txt:2020-07-24T14:29:31.456859-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:226) ActiveStream::transitionState: Transitioning from backfilling to in-memory memcached.log.000004.txt:2020-07-24T14:29:32.062066-07:00 INFO 758: (default) DCP (Producer) eq_dcpq:fts:social_234ec66318275839_f4e0a48a-33d8da21 - (vb:216) Backfill complete, 11326 items read from disk, 643 from memory, last seqno read: 95680, pendingBackfill : False memcached.log.000004.txt:2020-07-24T14:29
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-2788 contains kv_engine commit 8cec59e with commit message:
          MB-40543: Merge branch '6.5.1' into mad-hatter

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2788 contains kv_engine commit 8cec59e with commit message: MB-40543 : Merge branch '6.5.1' into mad-hatter
          Hide
          build-team Couchbase Build Team added a comment -

          Build couchbase-server-7.0.0-2788 contains kv_engine commit 166a75a with commit message:
          MB-40543: Add dynamic blacklist FTS log config option

          Show
          build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-2788 contains kv_engine commit 166a75a with commit message: MB-40543 : Add dynamic blacklist FTS log config option

            People

            Assignee:
            malarky Chris Malarky
            Reporter:
            malarky Chris Malarky
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Due:
              Created:
              Updated:
              Resolved:

                Gerrit Reviews

                There are no open Gerrit changes

                  PagerDuty