Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Critical
Fix Version/s: 7.0.0
Affects Version/s: Cheshire-Cat
Component/s: secondary-index
Labels:
- approved-for-7.0.0
- volume-test

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:

Hide
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.106.134.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.106.136.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.58.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.73.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.74.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.75.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.81.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.86.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.121.77.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.24.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.25.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.26.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.31.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.32.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.33.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.122.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.14.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.243.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.254.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.48.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.105.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.110.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.112.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.148.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.149.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.150.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.151.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.241.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.74.zip

Show
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.106.134.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.106.136.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.58.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.73.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.74.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.75.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.81.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.120.86.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.121.77.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.24.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.25.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.26.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.31.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.32.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.123.33.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.122.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.14.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.243.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.254.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.96.48.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.105.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.110.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.112.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.148.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.149.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.150.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.151.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.241.zip url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1621059171/collectinfo-2021-05-15T061303-ns_1%40172.23.97.74.zip
Story Points:
1
Is this a Regression?:
Unknown

Description

Build : 7.0.0-5161
Test : -test tests/2i/cheshirecat/test_idx_cc_vol_10K.yml -scope tests/2i/cheshirecat/scope_idx_cc_vol_10K.yml
Scale : 5
Iteration : 1st

While investigating ~~MB-46317~~, it was observed that the indexer logs on 172.23.123.31 were flooded with 1.15M+ msgs of the following type between 2021-05-14T21:55 & 2021-05-14T22:11 :

2021-05-14T21:55:27.598-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 293. CurrentTS: Snapshot 24192-24483 Seqno 24459 Vbuuid 63174434721350. LastFlushTS: Snapshot 24192-24483 Seqno 24483 Vbuuid 63174434721350.

2021-05-14T21:55:27.607-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket5 StreamId MAINT_STREAM vbucket 293. CurrentTS: Snapshot 24192-24483 Seqno 24459 Vbuuid 63174434721350. LastFlushTS: Snapshot 24192-24483 Seqno 24483 Vbuuid 63174434721350.

2021-05-14T21:55:27.607-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket1 StreamId MAINT_STREAM vbucket 9. CurrentTS: Snapshot 47366-47535 Seqno 47494 Vbuuid 8721681831084. LastFlushTS: Snapshot 47366-47535 Seqno 47535 Vbuuid 8721681831084.

2021-05-14T21:55:27.607-07:00 [Info] Timekeeper::ensureMonotonicTs  Align seqno smaller than lastFlushTs. KeyspaceId bucket1 StreamId MAINT_STREAM vbucket 11. CurrentTS: Snapshot 93751-93904 Seqno 93895 Vbuuid 228381830366397. LastFlushTS: Snapshot 93751-93904 Seqno 93904 Vbuuid 228381830366397.

This causes logs to rotate too fast and can become a supportability issue.

Attachments

Issue Links

relates to

MB-47302 [System Test]: Msg - "Timekeeper::ensureMonotonicTs Align seqno smaller than lastFlushTs" in indexer logs

Resolved

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Mihir Kamdar (Inactive)

Reporter:: Mihir Kamdar (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Due:: 28/May/21

Created:: 15/May/21 12:54 PM

Updated:: 09/Jul/21 12:35 AM

Resolved:: 27/May/21 5:38 PM

Gerrit Reviews

There are no open Gerrit changes

Show There are 4 closed Gerrit changes

Hide There are 4 closed Gerrit changes

MB-46318 skip logging info for disable align TS: Gerrit Review:

MB-46318 use config for reset empty snapshot index: Gerrit Review:

MB-46318 log more information with StreamBegin: Gerrit Review:

MB-46318 log more info for monotonic TS check: Gerrit Review:

[Volume Test] Indexer logs flooded with msg "Timekeeper::ensureMonotonicTs Align seqno smaller than lastFlushTs"

Details

Description

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty