Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45759

All timer tests with 512 bytes context failing with timer_msg_count = 1

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • clients
    • 7.0.0-4907
    • Triaged
    • 1
    • Unknown

    Description

      All 512byte context tests are failing with this flooding the eventing logs - looks like a JSON parsing issue.

       678\"}","vb":800}}
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_18:/tmp/127.0.0.1:8091_18_954300937.sock:92690] ReferenceKey json parse failed, ReferenceKey: eventing::954300937:tm:800:cx:HUsmYdlezRJAO*StVvNp*JqSr6F value :{"alr":"eventing::954300937:tm:800:al:1618643957:1488","ctx":{"callback":"timerCallback","context":"{\"docId\":\"eventing-000025096304\",\"field\":\"01234567890123456789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] 6789...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      2021-04-17T02:01:33.125-07:00 [Info] eventing-consumer [worker_perf-test1_4:/tmp/127.0.0.1:8091_4_954300937.sock:92392] ...truncated due to encoding error
      

       The stats on the eventing node are : 

       on_update_success: 50000000
       timer_create_counter: 50000000
       timer_callback_success : 0

      The timers were all created but - none of the callbacks were received.

      Eventing Rebalance time has also increased in a similar test by 3x : http://perf.jenkins.couchbase.com/job/themis/9831/console 

      Test Log : http://perf.jenkins.couchbase.com/job/themis/9924/consoleFull 

      Logs : 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.16.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.17.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.20.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.96.23.zip

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9924/172.23.97.177.zip - Eventing

       

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-7.0.0-5043 contains tlm commit 08f04e8 with commit message:
            MB-45759: Set new LCB_SNAPPY_xxx vars so libcouchbase uses our Snappy

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-5043 contains tlm commit 08f04e8 with commit message: MB-45759 : Set new LCB_SNAPPY_xxx vars so libcouchbase uses our Snappy

            Chris Hillery, could you check that now libcouchbase is using proper snappy?

            avsej Sergey Avseyev added a comment - Chris Hillery , could you check that now libcouchbase is using proper snappy?

            Abhishek Jindal Your toy build didn't include my tlm change, which is required to activate the libcouchbase change. Also, FYI, you can't specify Gerrit changes to the "manifest" repo to the toy-unix-simple job; I've updated the parameter description to make that explicit.

            Since both changes are already merged now, you can re-try your tests with 7.0.0-5043.

            Sergey Avseyev I confirm that in 7.0.0-5043, libcouchbase is linked against Server's own libsnappy.so.

            [root@d3605a6896d2 lib]# ldd libcouchbase.so.7
                linux-vdso.so.1 =>  (0x00007ffeecffe000)
                libdl.so.2 => /lib64/libdl.so.2 (0x00007fa526c17000)
                libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fa5269fd000)
                libssl.so.1.1 => /opt/couchbase/lib/./../lib/libssl.so.1.1 (0x00007fa526769000)
                libcrypto.so.1.1 => /opt/couchbase/lib/./../lib/libcrypto.so.1.1 (0x00007fa52625c000)
                librt.so.1 => /lib64/librt.so.1 (0x00007fa526054000)
                libsnappy.so.1 => /opt/couchbase/lib/./../lib/libsnappy.so.1 (0x00007fa525e4d000)
                libm.so.6 => /lib64/libm.so.6 (0x00007fa525b4b000)
                libz.so.1 => /opt/couchbase/lib/./../lib/libz.so.1 (0x00007fa52592e000)
                libstdc++.so.6 => /opt/couchbase/lib/./../lib/libstdc++.so.6 (0x00007fa525561000)
                libgcc_s.so.1 => /opt/couchbase/lib/./../lib/libgcc_s.so.1 (0x00007fa525349000)
                libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa52512d000)
                libc.so.6 => /lib64/libc.so.6 (0x00007fa524d5f000)
                /lib64/ld-linux-x86-64.so.2 (0x00007fa527156000)
            

            ceej Chris Hillery added a comment - Abhishek Jindal  Your toy build didn't include my tlm change, which is required to activate the libcouchbase change. Also, FYI, you can't specify Gerrit changes to the "manifest" repo to the toy-unix-simple job; I've updated the parameter description to make that explicit. Since both changes are already merged now, you can re-try your tests with 7.0.0-5043. Sergey Avseyev  I confirm that in 7.0.0-5043, libcouchbase is linked against Server's own libsnappy.so. [root @d3605a6896d2 lib]# ldd libcouchbase.so. 7 linux-vdso.so. 1 => ( 0x00007ffeecffe000 ) libdl.so. 2 => /lib64/libdl.so. 2 ( 0x00007fa526c17000 ) libresolv.so. 2 => /lib64/libresolv.so. 2 ( 0x00007fa5269fd000 ) libssl.so. 1.1 => /opt/couchbase/lib/./../lib/libssl.so. 1.1 ( 0x00007fa526769000 ) libcrypto.so. 1.1 => /opt/couchbase/lib/./../lib/libcrypto.so. 1.1 ( 0x00007fa52625c000 ) librt.so. 1 => /lib64/librt.so. 1 ( 0x00007fa526054000 ) libsnappy.so. 1 => /opt/couchbase/lib/./../lib/libsnappy.so. 1 ( 0x00007fa525e4d000 ) libm.so. 6 => /lib64/libm.so. 6 ( 0x00007fa525b4b000 ) libz.so. 1 => /opt/couchbase/lib/./../lib/libz.so. 1 ( 0x00007fa52592e000 ) libstdc++.so. 6 => /opt/couchbase/lib/./../lib/libstdc++.so. 6 ( 0x00007fa525561000 ) libgcc_s.so. 1 => /opt/couchbase/lib/./../lib/libgcc_s.so. 1 ( 0x00007fa525349000 ) libpthread.so. 0 => /lib64/libpthread.so. 0 ( 0x00007fa52512d000 ) libc.so. 6 => /lib64/libc.so. 6 ( 0x00007fa524d5f000 ) /lib64/ld-linux-x86- 64 .so. 2 ( 0x00007fa527156000 )

            Chris Hillery Yes, looks like it was a problem with the toy. Can confirm that the lcb (and eventing) can now decompress the document correctly. Resolving.

            abhishek.jindal Abhishek Jindal added a comment - Chris Hillery Yes, looks like it was a problem with the toy. Can confirm that the lcb (and eventing) can now decompress the document correctly. Resolving.
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) added a comment - verified for 7.0.0-5043 http://perf.jenkins.couchbase.com/job/themis/10075/

            People

              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty