Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48774

LCB error 1016 (LCB_ERROR_FLAG_TRANSIENT) in performance tests

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • Neo
    • Neo
    • eventing

    Description

      Build: 7.1.0-1374 , not seen on 7.1.0-1345

      Test: time100M_fun_bktop.test

      errors : 1016 ( Client not bootstrapped. Ensure bootstrap/connect was attempted and was successful)

      "lcb_exception_stats": {
                  "1016": 203
            } 

      Job: http://perf.jenkins.couchbase.com/job/themis/12917/console 

      Attachments

        For Gerrit Dashboard: MB-48774
        # Subject Branch Project Status CR V

        Activity

          LCB status 1016 specifies that the SDK bootstrap hasn't yet been completed.
          Looking at the logs, we do see that lcb bootstrap timed out against the first KV node after 4 seconds probably due to a network blip:

          2021-10-05T15:19:03.171-07:00 [Info] [lcb,instance L:504 I:1228723907] Effective connection string: couchbase://172.23.96.16:11210/eventing-bucket-1?select_bucket=true&detailed_errcodes=1. Bucket=eventing-bucket-1
          2021-10-05T15:19:03.173-07:00 [Info] [lcb,cccp L:176 I:1228723907] Requesting connection to node 172.23.96.16:11210 for CCCP configuration
          2021-10-05T15:19:05.175-07:00 [Info] [lcb,cccp L:187 I:1228723907] <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_ERR_TIMEOUT (201)
          2021-10-05T15:19:05.175-07:00 [Info] [lcb,confmon L:216 I:1228723907] Provider 'CCCP' failed: LCB_ERR_TIMEOUT (201)
          2021-10-05T15:19:07.179-07:00 [Info] [lcb,htconfig L:341 I:1228723907] <172.23.96.16:8091> (CTX=0x7fc82c0efb80,bc_http) HTTP Provider timed out waiting for I/O
          

          This could've been avoided as there are 4 more KV nodes in the cluster this LCB instance could've successfully bootstrapped with.
          https://review.couchbase.org/c/eventing/+/166500 addresses this where we will now attempt a bootstrap from all KV nodes in the cluster before giving up.

          abhishek.jindal Abhishek Jindal added a comment - LCB status 1016 specifies that the SDK bootstrap hasn't yet been completed. Looking at the logs, we do see that lcb bootstrap timed out against the first KV node after 4 seconds probably due to a network blip: 2021-10-05T15:19:03.171-07:00 [Info] [lcb,instance L:504 I:1228723907] Effective connection string: couchbase://172.23.96.16:11210/eventing-bucket-1?select_bucket=true&detailed_errcodes=1. Bucket=eventing-bucket-1 2021-10-05T15:19:03.173-07:00 [Info] [lcb,cccp L:176 I:1228723907] Requesting connection to node 172.23.96.16:11210 for CCCP configuration 2021-10-05T15:19:05.175-07:00 [Info] [lcb,cccp L:187 I:1228723907] <NOHOST:NOPORT> (CTX=(nil),) Could not get configuration: LCB_ERR_TIMEOUT (201) 2021-10-05T15:19:05.175-07:00 [Info] [lcb,confmon L:216 I:1228723907] Provider 'CCCP' failed: LCB_ERR_TIMEOUT (201) 2021-10-05T15:19:07.179-07:00 [Info] [lcb,htconfig L:341 I:1228723907] <172.23.96.16:8091> (CTX=0x7fc82c0efb80,bc_http) HTTP Provider timed out waiting for I/O This could've been avoided as there are 4 more KV nodes in the cluster this LCB instance could've successfully bootstrapped with. https://review.couchbase.org/c/eventing/+/166500 addresses this where we will now attempt a bootstrap from all KV nodes in the cluster before giving up.

          Build couchbase-server-7.1.0-1806 contains eventing commit 1b0c37f with commit message:
          MB-48774 : Add all KV nodes to LCB bootstrap list

          build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1806 contains eventing commit 1b0c37f with commit message: MB-48774 : Add all KV nodes to LCB bootstrap list

          Not seen on 7.1.0-1831

          vikas.chaudhary Vikas Chaudhary added a comment - Not seen on 7.1.0-1831

          People

            vikas.chaudhary Vikas Chaudhary
            vikas.chaudhary Vikas Chaudhary
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty