Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53263

[System Test] [6.6.5-10108]Disk commit timestamp is not snapshot aligned

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Duplicate
    • 6.6.5
    • None
    • secondary-index
    • None
    • Version: Enterprise Edition 6.6.5 build 10108
    • Untriaged
    • 1
    • Unknown

    Description

      We've had a hit on the disk snapshot alignment issue in our longevity test run. 

       

      172.23.104.137 : index
      2022-08-04T06:47:21.605-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
      2022-08-04T06:47:53.192-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
      2022-08-04T07:06:36.540-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
      2022-08-04T07:07:10.967-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
       
      172.23.104.69 : index
      2022-08-04T06:56:42.390-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
       
      172.23.105.168 : index
      2022-08-04T07:26:19.189-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
       
      172.23.120.245 : index
      2022-08-04T06:36:59.408-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
      2022-08-04T06:37:33.050-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
      2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -
       
      172.23.96.252 : index
      2022-08-04T06:24:52.526-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
       
      172.23.96.253 : index
      2022-08-04T06:47:20.669-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
       

       

      CBCollect -> 

      cbcollect logs: 
       
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.104.155.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.104.157.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.104.5.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.104.67.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.104.69.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.105.107.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.105.111.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.105.168.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.106.188.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.108.103.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.120.107.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.120.245.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.121.117.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.123.28.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.96.148.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.96.192.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.96.252.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.96.253.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.97.119.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.97.121.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.97.122.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.97.239.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.99.11.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.99.20.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.99.21.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659624401/collectinfo-2022-08-04T144643-ns_1%40172.23.99.25.zip 

      cc Balakumaran Gopal Ritam Sharma Amit Kulkarni Deepkaran Salooja 

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            varun.velamuri Varun Velamuri added a comment - - edited

            Pavan PB The warnings for INIT_STREAM can be ignored as indexer can persist a non-snapshot aligned timestamp for INIT_STREAM. https://review.couchbase.org/c/indexing/+/178488 will fix the assertion for INIT_STREAM

            But, this message is concerning as snapshot alignment is missing for MAINT_STREAM

            2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -
            

             

             

            varun.velamuri Varun Velamuri added a comment - - edited Pavan PB The warnings for INIT_STREAM can be ignored as indexer can persist a non-snapshot aligned timestamp for INIT_STREAM. https://review.couchbase.org/c/indexing/+/178488 will fix the assertion for INIT_STREAM But, this message is concerning as snapshot alignment is missing for MAINT_STREAM 2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -    
            pavan.pb Pavan PB added a comment -

            We don't seem to have logs for one of the indexer nodes as noted by Varun. But it happened on the next iteration as well. Here are the logs 

             

            cbcollect logs:

                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.137.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.155.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.157.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.5.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.67.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.69.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.105.107.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.105.111.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.105.168.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.106.100.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.106.188.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.108.103.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.120.107.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.120.245.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.123.28.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.148.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.192.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.252.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.253.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.119.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.121.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.122.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.239.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.11.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.20.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.21.zip
                     url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.25.zip

             

            172.23.104.69 : index
            2022-08-04T06:56:42.390-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.105.168 : index
            2022-08-04T07:26:19.189-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T07:34:15.647-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.120.245 : index
            2022-08-04T06:36:59.408-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T06:37:33.050-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -
             
            172.23.96.252 : index
            2022-08-04T06:24:52.526-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.96.253 : index
            2022-08-04T06:47:20.669-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T07:34:16.877-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.104.69 : index
            2022-08-04T06:56:42.390-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.105.168 : index
            2022-08-04T07:26:19.189-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T07:34:15.647-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.120.245 : index
            2022-08-04T06:36:59.408-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T06:37:33.050-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -
             
            172.23.96.252 : index
            2022-08-04T06:24:52.526-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             
            172.23.96.253 : index
            2022-08-04T06:47:20.669-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
            2022-08-04T07:34:16.877-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default, tsVbuuid: bucket: default, vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -
             

            pavan.pb Pavan PB added a comment - We don't seem to have logs for one of the indexer nodes as noted by Varun. But it happened on the next iteration as well. Here are the logs    cbcollect logs:          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.137.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.155.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.157.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.5.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.67.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.104.69.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.105.107.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.105.111.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.105.168.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.106.100.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.106.188.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.108.103.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.120.107.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.120.245.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.123.28.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.148.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.192.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.252.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.96.253.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.119.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.121.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.122.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.97.239.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.11.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.20.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.21.zip          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1659627297/collectinfo-2022-08-04T153459-ns_1%40172.23.99.25.zip   172.23 . 104.69 : index 2022 - 08 -04T06: 56 : 42.390 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 105.168 : index 2022 - 08 -04T07: 26 : 19.189 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T07: 34 : 15.647 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 120.245 : index 2022 - 08 -04T06: 36 : 59.408 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T06: 37 : 33.050 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T06: 37 : 59.515 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -   172.23 . 96.252 : index 2022 - 08 -04T06: 24 : 52.526 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 96.253 : index 2022 - 08 -04T06: 47 : 20.669 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T07: 34 : 16.877 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 104.69 : index 2022 - 08 -04T06: 56 : 42.390 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 105.168 : index 2022 - 08 -04T07: 26 : 19.189 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T07: 34 : 15.647 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 120.245 : index 2022 - 08 -04T06: 36 : 59.408 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T06: 37 : 33.050 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T06: 37 : 59.515 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -   172.23 . 96.252 : index 2022 - 08 -04T06: 24 : 52.526 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -   172.23 . 96.253 : index 2022 - 08 -04T06: 47 : 20.669 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP - 2022 - 08 -04T07: 34 : 16.877 - 07 : 00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: INIT_STREAM, Bucket: default , tsVbuuid: bucket: default , vbuckets: 1024 Crc64: 3185084385719423624 snapType DISK_SNAP -

            a. Taking node 120.245 as example, vb:1018 & vb:1022 has mis-aligned timestamps

             

            2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -
                {vbno, vbuuid, seqno, snapshot-start, snapshot-end}
              { 1018     86684a1d33c8       4174       4011       4192}
                { 1019      d52e7685b88       4147       3963       4147}
                { 1020     7127bb1401ec       4168       3988       4168}
                { 1021      6895bc24038       4156       3975       4156}
                { 1022     4f2a2c2f1418       3984       3881       4036}
            

            b. Indexer generated DISK_SNAP because of build completion TS

            2022-08-04T06:37:59.439-07:00 [Info] timekeeper.repairMissingStreamBegin stream MAINT_STREAM done
            2022-08-04T06:37:59.515-07:00 [Info] timekeeper::generateNewStability: setting snapshot type as DISK_SNAP due to BuildCompletionTS

            When index builds for first time, it gets builds in MAINT_STREAM. In such a case, the DISK_SNAP need not be snapshot aligned. Also, the build completion timestamp, the indexer timestamp need not be snapshot aligned. Indexer is hitting that case here

             

            varun.velamuri Varun Velamuri added a comment - a. Taking node 120.245 as example, vb:1018 & vb:1022 has mis-aligned timestamps   2022-08-04T06:37:59.515-07:00 [Fatal] StorageMgr::handleCreateSnapshot Disk commit timestamp is not snapshot aligned. Stream: MAINT_STREAM, Bucket: CUSTOMER, tsVbuuid: bucket: CUSTOMER, vbuckets: 1024 Crc64: 849771906969854749 snapType DISK_SNAP -     {vbno, vbuuid, seqno, snapshot-start, snapshot-end}   { 1018     86684a1d33c8       4174       4011       4192}     { 1019      d52e7685b88       4147       3963       4147}     { 1020     7127bb1401ec       4168       3988       4168}     { 1021      6895bc24038       4156       3975       4156}     { 1022     4f2a2c2f1418       3984       3881       4036} b. Indexer generated DISK_SNAP because of build completion TS 2022-08-04T06:37:59.439-07:00 [Info] timekeeper.repairMissingStreamBegin stream MAINT_STREAM done 2022-08-04T06:37:59.515-07:00 [Info] timekeeper::generateNewStability: setting snapshot type as DISK_SNAP due to BuildCompletionTS When index builds for first time, it gets builds in MAINT_STREAM. In such a case, the DISK_SNAP need not be snapshot aligned. Also, the build completion timestamp, the indexer timestamp need not be snapshot aligned. Indexer is hitting that case here  

            Lynn Straus - This issue is a consequence of assertion check we have added in MB-53183. I have fixed the check in MB-53183. Closing this as duplicate of MB-53183.

            varun.velamuri Varun Velamuri added a comment - Lynn Straus - This issue is a consequence of assertion check we have added in MB-53183 . I have fixed the check in MB-53183 . Closing this as duplicate of MB-53183 .

            Fixed in MB-53183

            varun.velamuri Varun Velamuri added a comment - Fixed in MB-53183

            People

              varun.velamuri Varun Velamuri
              pavan.pb Pavan PB
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty