Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-46651

[System Test] : Messages "Timekeeper::ensureMonotonicTs Align seqno smaller than lastFlushTs" seen in the indexer logs

    XMLWordPrintable

Details

    Description

      Build : 7.0.0-5247 (RC3)
      Test : -test tests/2i/cheshirecat/test_idx_clusterops_cheshire_cat_recovery.yml -scope tests/2i/cheshirecat/scope_idx_cheshire_cat_dgm.yml (Plasma Recovery)
      Scale : 2
      Iteration : 1st

      In the indexer logs on 172.23.107.3, 172.23.97.216, 172.23.97.217, following messages can be seen :

      172.23.107.3 : index
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz:2021-05-28T18:19:56.592-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 466. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 15564553833407. LastFlushTS: Snapshot 196-212 Seqno 212 Vbuuid 15564553833407. HWT [196-212, 212]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.2.gz:2021-05-28T18:19:56.608-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 468. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 125632719904058. LastFlushTS: Snapshot 196-212 Seqno 212 Vbuuid 125632719904058. HWT [196-212, 212]. LastSnap [196-196, 196].
       
      172.23.97.216 : index
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-05-28T18:57:03.022-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 938. CurrentTS: Snapshot 271-271 Seqno 271 Vbuuid 143222648148282. LastFlushTS: Snapshot 271-278 Seqno 278 Vbuuid 143222648148282. HWT [271-278, 278]. LastSnap [271-271, 271].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-05-28T18:57:03.097-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 974. CurrentTS: Snapshot 271-271 Seqno 271 Vbuuid 135864086697527. LastFlushTS: Snapshot 271-278 Seqno 278 Vbuuid 135864086697527. HWT [271-278, 278]. LastSnap [271-271, 271].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log:2021-05-28T18:57:03.097-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 975. CurrentTS: Snapshot 271-271 Seqno 271 Vbuuid 39585727127243. LastFlushTS: Snapshot 271-278 Seqno 278 Vbuuid 39585727127243. HWT [271-278, 278]. LastSnap [271-271, 271].
       
      172.23.97.217 : index
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:57.949-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 440. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 250567766458491. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 250567766458491. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.518-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 497. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 119313135960709. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 119313135960709. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.670-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 497. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 119313135960709. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 119313135960709. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.670-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 899. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 200582711942381. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 200582711942381. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.670-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 903. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 268768421659204. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 268768421659204. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.670-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 911. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 133636183445792. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 133636183445792. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.743-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 669. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 212187489382173. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 212187489382173. HWT [196-208, 208]. LastSnap [196-196, 196].
      /opt/couchbase/var/lib/couchbase/logs/indexer.log.1.gz:2021-05-28T18:17:58.754-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 669. CurrentTS: Snapshot 196-196 Seqno 196 Vbuuid 212187489382173. LastFlushTS: Snapshot 196-208 Seqno 208 Vbuuid 212187489382173. HWT [196-208, 208]. LastSnap [196-196, 196].
      

      These messages were reported first in MB-46318. In this test run, these msgs are not flooding the test log, hence the issue is not critical. But the ticket has been opened as advised post fixing in MB-46318.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            mihir.kamdar Mihir Kamdar (Inactive)
            mihir.kamdar Mihir Kamdar (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty