Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45759

All timer tests with 512 bytes context failing with timer_msg_count = 1

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • clients
    • 7.0.0-4907
    • Triaged
    • 1
    • Unknown

    Description

      All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.

       678\"}","vb":800}}
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      

       The stats on the eventing node are : 

       on_update_success: 50000000
       timer_create_counter: 50000000
       timer_callback_success : 0

      The timers were all created but - none of the callbacks were received.

      Eventing Rebalance time has also increased in a similar test by 3x : http://perf.jenkins.couchbase.com/job/themis/9831/console 

      Test Log : http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull 

      Logs : 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip - Eventing

       

      Attachments

        Issue Links

          Activity

            People

              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty