Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44784

[Volume Test]: Timers not getting fired

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0 build 4619 

      Test: Eventing Volume

      • Create 9 node cluster ( Eventing:3 , Kv:4,index:1,query:1)
      • Create 15 handlers (3 of bucket op, timer,n1ql, SBM, curl)
      • deploy bucket op, timers and N1ql
      • Rebalance in 1 eventing node
      • Start CRUD on bucket_op where no handler are deployed
      • Verify handlers
      • Load more data
      • Add to 2 KV nodes
      • deploy curl and SBM handlers
      • Rebalance out 2 kv nodes
      • Verify all handlers
      • Pause bucket op and timers
      • Swap 2 kv nodes 1 after another
      • load more data
      • Rebalance in 2 eventing nodes
      • Resume bucket op and timers
      • Verify all handlers
      • load more data
      • rebalance out 2 eventing nodes
      • verify all handlers
      • load more data
      • Swap 2 eventing nodes 1 after another
      • Verify all handlers
      • load all data
      • Rebalance in kv and eventing node
      • Verify all handlers
      • load all data
      • Rebalance out 1 kv 1 eventing
      • Verify all handlers
      • load all data
      • Swap 1 kv and 1 eventing
      • Verify all handlers

      Observed that all verification for timer handler failed due to data miss match and seeing the documents in metadata

      select count(*) from metadata.scope_1.coll0 where 
      meta().id like 'eventing::%:sp' and sta != stp; --> 128
       
      select count(*) from metadata.scope_1.coll1 where meta().id like 'eventing::%:sp' and sta != stp; --> 128
       
      select count(*) from metadata.scope_1.coll2 where meta().id like 'eventing::%:sp' and sta != stp; --> 128
      
      

       

      Handler Source Scope Destination Scope
      timers0_0 timer_op.scope_0.coll0(4095000) timer_op.scope_1.coll0(4094498)
      timers1_0 timer_op.scope_0.coll1(4091100) timer_op.scope_1.coll1(4089924)
      timers2_0 timer_op.scope_0.coll2(4089600) timer_op.scope_1.coll2(4089127)

       

      N1ql queries failing with CAS miss match

      2021-03-05T13:20:27.900-08:00 [INFO] "Query failed: " {"message":"SDK error : LCB_ERR_CAS_MISMATCH (209) Query error : {\n\"requestID\": \"1bc36320-7bb6-4e5f-a47d-d33f7c235c62\",\n\"clientContextID\": \"4@n1ql0_0.js(OnUpdate)\",\n\"signature\": null,\n\"results\": [\n],\n\"errors\": [{\"code\":12009,\"msg\":\"DML Error, possible causes include CAS mismatch or concurrent modificationFailed to perform UPSERT - cause: dial tcp 172.23.105.25:11210: connect: cannot assign requested address\"}],\n\"status\": \"errors\",\n\"metrics\": {\"elapsedTime\": \"23.135053ms\",\"executionTime\": \"23.080655ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 15,\"errorCount\": 1}\n}\n","stack":"Error\n    at N1QL (<anonymous>)\n    at OnUpdate (n1ql0_0.js:4:21)"} 2021-03-05T13:20:27.875-08:00 [INFO] "Query failed: " {"message":"SDK error : LCB_ERR_CAS_MISMATCH (209) Query error : {\n\"requestID\": \"1570ce91-f811-469c-b4b3-0ac8f1b57cca\",\n\"clientContextID\": \"4@n1ql0_0.js(OnUpdate)\",\n\"signature\": null,\n\"results\": [\n],\n\"errors\": [{\"code\":12009,\"msg\":\"DML Error, possible causes include CAS mismatch or concurrent modificationFailed to perform UPSERT - cause: dial tcp 172.23.105.25:11210: connect: cannot assign requested address\"}],\n\"status\": \"errors\",\n\"metrics\": {\"elapsedTime\": \"18.964561ms\",\"executionTime\": \"18.922147ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 9,\"errorCount\": 1}\n}\n","stack":"Error\n    at N1QL (<anonymous>)\n    at OnUpdate (n1ql0_0.js:4:21)"}  

      Attachments

        1. 693280967_timer2.log
          273 kB
        2. MB-44784.rtf
          34 kB
        3. metadata_items.png
          metadata_items.png
          452 kB
        4. metadata_items-1.png
          metadata_items-1.png
          452 kB
        5. metadata_scope1_coll1_1.png
          metadata_scope1_coll1_1.png
          352 kB
        6. metadata_scope1_coll1_2.png
          metadata_scope1_coll1_2.png
          318 kB
        7. metadata_scope1_coll1_3.png
          metadata_scope1_coll1_3.png
          300 kB
        8. metadata_scope1_coll1_4.png
          metadata_scope1_coll1_4.png
          301 kB
        9. RunLogs.txt
          239 kB
        10. Screenshot 2021-03-12 at 1.41.58 PM.png
          Screenshot 2021-03-12 at 1.41.58 PM.png
          70 kB
        11. Screenshot 2021-03-12 at 3.23.20 PM.png
          Screenshot 2021-03-12 at 3.23.20 PM.png
          153 kB
        12. Screenshot 2021-04-08 at 11.32.59 AM.png
          Screenshot 2021-04-08 at 11.32.59 AM.png
          606 kB
        13. timer_2 metadata.zip
          3.59 MB
        14. timer_mismatch.png
          timer_mismatch.png
          356 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            vikas.chaudhary Vikas Chaudhary created issue -
            ritam.sharma Ritam Sharma made changes -
            Field Original Value New Value
            Summary Volume: Timers not getting fired [Volume Test]: Timers not getting fired
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Abhishek Jindal [ abhishek.jindal ]
            vikas.chaudhary Vikas Chaudhary made changes -
            vikas.chaudhary Vikas Chaudhary made changes -
            Attachment backup_metadaa.zip [ 129842 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment timer_2 metadata.zip [ 130648 ]
            abhishek.jindal Abhishek Jindal made changes -
            abhishek.jindal Abhishek Jindal made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment 693280967_timer2.log [ 131137 ]
            abhishek.jindal Abhishek Jindal made changes -
            Component/s clients [ 10042 ]
            Component/s eventing [ 14026 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Matt Ingenthron [ ingenthr ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Matt Ingenthron [ ingenthr ] Michael Nitschinger [ daschl ]
            jeelan.poola Jeelan Poola made changes -
            Labels volume-test affects-cc-testing volume-test
            ingenthr Matt Ingenthron made changes -
            Assignee Michael Nitschinger [ daschl ] Sergey Avseyev [ avsej ]
            avsej Sergey Avseyev made changes -
            Assignee Sergey Avseyev [ avsej ] Jeelan Poola [ jeelan.poola ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Ankit Prabhu [ ankit.prabhu ]
            ankit.prabhu Ankit Prabhu made changes -
            Assignee Ankit Prabhu [ ankit.prabhu ] Vikas Chaudhary [ vikas.chaudhary ]
            vikas.chaudhary Vikas Chaudhary made changes -
            Assignee Vikas Chaudhary [ vikas.chaudhary ] Abhishek Jindal [ abhishek.jindal ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Vikas Chaudhary [ vikas.chaudhary ]
            ritam.sharma Ritam Sharma made changes -
            Assignee Vikas Chaudhary [ vikas.chaudhary ] Jeelan Poola [ jeelan.poola ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Sergey Avseyev [ avsej ]
            pablo.silberkasten Pablo Silberkasten (Inactive) made changes -
            Assignee Sergey Avseyev [ avsej ] Jeelan Poola [ jeelan.poola ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Pablo Silberkasten [ JIRAUSER25235 ]
            pablo.silberkasten Pablo Silberkasten (Inactive) made changes -
            Assignee Pablo Silberkasten [ JIRAUSER25235 ] Vikas Chaudhary [ vikas.chaudhary ]
            ritam.sharma Ritam Sharma made changes -
            Assignee Vikas Chaudhary [ vikas.chaudhary ] Pablo Silberkasten [ JIRAUSER25235 ]
            arunkumar Arunkumar Senthilnathan made changes -
            Assignee Pablo Silberkasten [ JIRAUSER25235 ] Abhishek Jindal [ abhishek.jindal ]
            jeelan.poola Jeelan Poola made changes -
            Component/s eventing [ 14026 ]
            Component/s clients [ 10042 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment metadata_scope1_coll1_1.png [ 134635 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment metadata_scope1_coll1_2.png [ 134636 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment metadata_scope1_coll1_3.png [ 134637 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment metadata_scope1_coll1_4.png [ 134638 ]
            abhishek.jindal Abhishek Jindal made changes -
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Vikas Chaudhary [ vikas.chaudhary ]
            ritam.sharma Ritam Sharma made changes -
            Assignee Vikas Chaudhary [ vikas.chaudhary ] Sujay Gad [ JIRAUSER25279 ]
            sujay.gad Sujay Gad made changes -
            Assignee Sujay Gad [ JIRAUSER25279 ] Abhishek Jindal [ abhishek.jindal ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sujay Gad [ JIRAUSER25279 ]
            ritam.sharma Ritam Sharma made changes -
            Assignee Sujay Gad [ JIRAUSER25279 ] Abhishek Jindal [ abhishek.jindal ]
            ritam.sharma Ritam Sharma made changes -
            Attachment MB-44784.rtf [ 135057 ]
            abhishek.jindal Abhishek Jindal made changes -
            Link This issue depends on MB-45599 [ MB-45599 ]
            sujay.gad Sujay Gad made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            sujay.gad Sujay Gad made changes -
            Link This issue relates to MB-45713 [ MB-45713 ]
            abhishek.jindal Abhishek Jindal made changes -
            Link This issue relates to MB-45713 [ MB-45713 ]
            sujay.gad Sujay Gad made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Vinayaka Kamath [ JIRAUSER25010 ]
            jeelan.poola Jeelan Poola made changes -
            Link This issue depends on MB-44324 [ MB-44324 ]
            lynn.straus Lynn Straus made changes -
            Due Date 27/Apr/21
            jeelan.poola Jeelan Poola made changes -
            Assignee Vinayaka Kamath [ JIRAUSER25010 ] Abhishek Jindal [ abhishek.jindal ]
            jeelan.poola Jeelan Poola made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            vikas.chaudhary Vikas Chaudhary made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            sujay.gad Sujay Gad made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            abhishek.jindal Abhishek Jindal made changes -
            Due Date 27/Apr/21 07/May/21
            abhishek.jindal Abhishek Jindal made changes -
            Attachment timer_mismatch.png [ 139425 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment metadata_items.png [ 139428 ]
            abhishek.jindal Abhishek Jindal made changes -
            Attachment metadata_items-1.png [ 139429 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sujay Gad [ JIRAUSER25279 ]
            abhishek.jindal Abhishek Jindal made changes -
            Due Date 07/May/21 14/May/21
            raju Raju Suravarjjala made changes -
            Due Date 14/May/21 10/May/21
            ritam.sharma Ritam Sharma made changes -
            Attachment RunLogs.txt [ 139949 ]
            ritam.sharma Ritam Sharma made changes -
            Assignee Sujay Gad [ JIRAUSER25279 ] Abhishek Jindal [ abhishek.jindal ]
            abhishek.jindal Abhishek Jindal made changes -
            Due Date 10/May/21 14/May/21
            abhishek.jindal Abhishek Jindal made changes -
            Link This issue relates to MB-45967 [ MB-45967 ]
            abhishek.jindal Abhishek Jindal made changes -
            Resolution Duplicate [ 3 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            mihir.kamdar Mihir Kamdar (Inactive) made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.0 [ 17233 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s Cheshire-Cat [ 15915 ]

            People

              abhishek.jindal Abhishek Jindal
              vikas.chaudhary Vikas Chaudhary
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty