Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47534

Unexpected duplicate SEQNO_ADVANCED from DCP producer

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Major
    • 7.0.2
    • 7.0.0, 7.0.1
    • couchbase-bucket
    • system test- tests/integration/cheshirecat/test_cheshirecat_kv_gsi_coll_xdcr_backup_sgw_fts_itemct_txns_eventing_cbas_scale3.yml -scope tests/integration/cheshirecat/scope_cheshirecat_with_backup.yml

    Description

      7.0.0-5247

      Observed while looking at logs for MB-46666, lots of (633171 to be exact) duplicate SEQNO_ADVANCED messages are received during the course of the test.
      e.g. a subset (due to log rollover) from one of the analytics nodes, shows ~30K redundant advancements on three vbuckets:

      $ cat sorted | uniq -c | sort -n | tail -4
      		 
         1 (Local/bucket7)[23]:BO] seqno advanced to 9735 for vbid 311 on DCP_SEQNO_ADVANCED
      		 
      9804 (Local/WAREHOUSE)[18]:BO] seqno advanced to 38653 for vbid 474 on DCP_SEQNO_ADVANCED
      		 
      10102 (Local/WAREHOUSE)[20]:BO] seqno advanced to 38404 for vbid 572 on DCP_SEQNO_ADVANCED
      		 
      10115 (Local/WAREHOUSE)[18]:BO] seqno advanced to 38736 for vbid 642 on DCP_SEQNO_ADVANCED

      If consumers should be prepared and expect duplicate SEQNOs_ADVANCED messages, we can accommodate that, but looking at the protocol documentation, the intent of the directive is "to indicate that the vbucket seqno has advanced for some event that the client is not subscribed to.", it doesn't seem to make much sense to me that duplicates should be expected.

      In the attached logs, the KV nodes are as follows:

           {{node,'ns_1@172.23.97.74',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,15}}},
      		 
           {{node,'ns_1@172.23.97.112',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,26221}}},
      		 
           {{node,'ns_1@172.23.123.24',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,58}}},
      		 
           {{node,'ns_1@172.23.97.241',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,24}}},
      		 
           {{node,'ns_1@172.23.120.86',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,86}}},
      		 
           {{node,'ns_1@172.23.123.26',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,72}}},
      		 
           {{node,'ns_1@172.23.120.77',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,80}}},
      		 
           {{node,'ns_1@172.23.123.25',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,66}}},
      		 
           {{node,'ns_1@172.23.120.73',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,44}}},
      		 
           {{node,'ns_1@172.23.96.14',services},{[kv],{<<"1cdab1a8387640a8210d5eab8468b344">>,35714}}},

      Attachments

        Issue Links

          is triggered by

          Bug - A problem which impairs or prevents the functions of the product. MB-46666 [System Test][Analytics] ERRO CBAS.util.ResourceLeakDetector [nioEventLoopGroup-2-8] LEAK: ByteBuf.release() was not called before it's garbage-collected

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-47009 [System Test] Index building stuck for 1 index

          • Major - Major loss of function.
          • Closed
          For Gerrit Dashboard: MB-47534
          # Subject Branch Project Status CR V
          159018,10 MB-47009: When resuming a filtered snapshot, check we reach the end cheshire-cat kv_engine Status: MERGED +2 +1

          Activity

            People

              owend Daniel Owen
              michael.blow Michael Blow
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty