Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44024

Timer creation failure due to LCB_ERR_TIMEOUT (201)

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • Cheshire-Cat
    • 7.0.0
    • eventing
    • 7.0.0-4342
    • Triaged
    • 1
    • Unknown
    • KV-Engine 2021-March

    Description

      We are seeing the following error in most of the timer based tests - 

      eventing-consumer [worker_perf-test1_2:/tmp/127.0.0.1:8091_2_4121979234.sock:194805] Timer Creation failed with message: Set Timer Failed with KV Error 2021-02-01T07:18:15.326-08:00 [Info] eventing-consumer [worker_perf-test1_2:/tmp/127.0.0.1:8091_2_4121979234.sock:194805] file : /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/eventing/v8_consumer/src/v8worker.cc line : 870 caller : SendUpdate : Returning empty valueError executing on_doc_update 2021-02-01T07:18:15.327-08:00 [Info] eventing-consumer [worker_perf-test1_21:/tmp/127.0.0.1:8091_21_4121979234.sock:195113] Exception: { "message": { "code": 1, "desc": "LCB_ERR_TIMEOUT (201): The request was not completed by the user-defined timeout", "name": "LCB_ERR_TIMEOUT" }, "stack": "Error\n at OnUpdate (perf-test1.js:6:5)"} Location: perf-test1.js:6 Code: createTimer(timerCallback, fireAt, meta.id, context); Stack: Error at OnUpdate (perf-test1.js:6:5) 2021-02-01T07:18:15.327-08:00 [Info] eventing-consumer [worker_perf-test1_2:/tmp/127.0.0.1:8091_2_4121979234.sock:194805] Exception: { "message": { "code": 1, "desc": "LCB_ERR_TIMEOUT (201): The request was not completed by the user-defined timeout", "name": "LCB_ERR_TIMEOUT" }, "stack": "Error\n at OnUpdate (perf-test1.js:6:5)"} Location: perf-test1.js:6 Code: createTimer(timerCallback, fireAt, meta.id, context); Stack: Error at OnUpdate (perf-test1.js:6:5) 2021-02-01T07:18:18.129-08:00 [Info] DCPT[eventing:nTq05Ae8-4:{eventing:nTq05Ae8-3:worker_perf-test1_0_172.23.96.16:11210_172.23.97.177:8096}/0] DCP-socket -> eventing blocked 100.733852ms (0.899432%)   
      

      Logs for the job are attached in the test here : http://perf.jenkins.couchbase.com/job/themis/8575/console  

      Eventing Logs : https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-8575/172.23.97.177.zip 

      "timer_create_failure": 1"timer_create_counter": 49999999
      

      Attachments

        Issue Links

          For Gerrit Dashboard: MB-44024
          # Subject Branch Project Status CR V

          Activity

            jeelan.poola Jeelan Poola added a comment -

            Thank you Daniel Owen!

            Matt Ingenthron Given above is not to be considered a rare occurrence, how should we to test & track the possibility of existing customer applications running into similar/same timeout issues with default 2.5s SDK timeout configurations when they upgrade to 7.0?

            Cc Arunkumar Senthilnathan FYI/FYA. Thank you!

            jeelan.poola Jeelan Poola added a comment - Thank you Daniel Owen ! Matt Ingenthron Given above is not to be considered a rare occurrence, how should we to test & track the possibility of existing customer applications running into similar/same timeout issues with default 2.5s SDK timeout configurations when they upgrade to 7.0? Cc Arunkumar Senthilnathan FYI/FYA. Thank you!

            Build couchbase-server-7.0.0-4701 contains eventing commit 72a063c with commit message:
            MB-44024: Make lcb_timeout configurable

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4701 contains eventing commit 72a063c with commit message: MB-44024 : Make lcb_timeout configurable

            Build couchbase-server-7.0.0-4701 contains eventing-ee commit 79dd36f with commit message:
            MB-44024: Make lcb_timeout configurable

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.0-4701 contains eventing-ee commit 79dd36f with commit message: MB-44024 : Make lcb_timeout configurable

            Logged MB-45125 to track SDK impact. Resolving this ticket as eventing side of the fixes are all done.

            jeelan.poola Jeelan Poola added a comment - Logged MB-45125 to track SDK impact. Resolving this ticket as eventing side of the fixes are all done.

            verified for 7.0.0-4735

            No timeouts seen in :  http://perf.jenkins.couchbase.com/job/themis/9457/ 

            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) added a comment - verified for 7.0.0-4735 No timeouts seen in :   http://perf.jenkins.couchbase.com/job/themis/9457/  

            People

              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty