Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45759

All timer tests with 512 bytes context failing with timer_msg_count = 1

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • clients
    • 7.0.0-4907
    • Triaged
    • 1
    • Unknown

    Description

      All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.

       678\"}","vb":800}}
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      

       The stats on the eventing node are : 

       on_update_success: 50000000
       timer_create_counter: 50000000
       timer_callback_success : 0

      The timers were all created but - none of the callbacks were received.

      Eventing Rebalance time has also increased in a similar test by 3x : http://perf.jenkins.couchbase.com/job/themis/9831/console 

      Test Log : http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull 

      Logs : 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip - Eventing

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) created issue -
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Field Original Value New Value
            Description All 512 context tests are failing with this flooding the eventing logs : 

             
            {noformat}
             678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             

            on_update_success: 50000000
            timer_create_counter: 50000000
            timer_callback_success : 0
            Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            All 512 context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat}
             on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.


             Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Description All 512 context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat}
             on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.


             Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat} on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.

            Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Vinayaka Kamath [ JIRAUSER25010 ]
            jeelan.poola Jeelan Poola made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Vinayaka Kamath [ JIRAUSER25010 ] Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ]
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Description All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat} on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.

            Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat} on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.

            Rebalance time has also increased in a similar test by 3x : [http://perf.jenkins.couchbase.com/job/themis/9831/console

            Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Description All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat} on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.

            Rebalance time has also increased in a similar test by 3x : [http://perf.jenkins.couchbase.com/job/themis/9831/console

            Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.
            {noformat} 678\"}","vb":800}}
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
            {noformat}
             The stats on the eventing node are : 
            {noformat} on_update_success: 50000000
             timer_create_counter: 50000000
             timer_callback_success : 0{noformat}
            The timers were all created but - none of the callbacks were received.

            Eventing Rebalance time has also increased in a similar test by 3x : [http://perf.jenkins.couchbase.com/job/themis/9831/console

            Test Log : [http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull

            Logs : 

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip]

            [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip] - Eventing

             
            jeelan.poola Jeelan Poola made changes -
            Due Date 22/Apr/21
            jeelan.poola Jeelan Poola made changes -
            Assignee Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ] Abhishek Jindal [ abhishek.jindal ]
            abhishek.jindal Abhishek Jindal made changes -
            Component/s clients [ 10042 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sergey Avseyev [ avsej ]
            jeelan.poola Jeelan Poola made changes -
            Component/s eventing [ 14026 ]
            avsej Sergey Avseyev made changes -
            Assignee Sergey Avseyev [ avsej ] Abhishek Jindal [ abhishek.jindal ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sergey Avseyev [ avsej ]
            avsej Sergey Avseyev made changes -
            Assignee Sergey Avseyev [ avsej ] Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ]
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Assignee Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ] Sergey Avseyev [ avsej ]
            ceej Chris Hillery made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 21966 ]
            avsej Sergey Avseyev made changes -
            Assignee Sergey Avseyev [ avsej ] Chris Hillery [ ceej ]
            ceej Chris Hillery made changes -
            Assignee Chris Hillery [ ceej ] Sergey Avseyev [ avsej ]
            ceej Chris Hillery made changes -
            Assignee Sergey Avseyev [ avsej ] Chris Hillery [ ceej ]
            ceej Chris Hillery made changes -
            Assignee Chris Hillery [ ceej ] Sergey Avseyev [ avsej ]
            avsej Sergey Avseyev made changes -
            Assignee Sergey Avseyev [ avsej ] Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ]
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Assignee Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ] Jeelan Poola [ jeelan.poola ]
            ceej Chris Hillery made changes -
            Remote Link This issue links to "Page (Couchbase, Inc. Wiki)" [ 21985 ]
            jeelan.poola Jeelan Poola made changes -
            Assignee Jeelan Poola [ jeelan.poola ] Abhishek Jindal [ abhishek.jindal ]
            jeelan.poola Jeelan Poola made changes -
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Resolved [ 5 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ] Abhishek Jindal [ abhishek.jindal ]
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Sergey Avseyev [ avsej ]
            avsej Sergey Avseyev made changes -
            Assignee Sergey Avseyev [ avsej ] Chris Hillery [ ceej ]
            ceej Chris Hillery made changes -
            Assignee Chris Hillery [ ceej ] Abhishek Jindal [ abhishek.jindal ]
            abhishek.jindal Abhishek Jindal made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            abhishek.jindal Abhishek Jindal made changes -
            Assignee Abhishek Jindal [ abhishek.jindal ] Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ]
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
            Triage Untriaged [ 10351 ] Triaged [ 10350 ]
            Status Resolved [ 5 ] Closed [ 6 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s 7.0.0 [ 17233 ]
            lynn.straus Lynn Straus made changes -
            Fix Version/s Cheshire-Cat [ 15915 ]

            People

              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty