Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-45196

LCB_ERR_INDEX_FAILURE - In n1ql eventing tests causing n1ql_op_exception_count = 1

    XMLWordPrintable

Details

    • Task
    • Status: Closed
    • Major
    • Resolution: Cannot Reproduce
    • Cheshire-Cat
    • 7.0.0
    • test-execution
    • 7.0.0-4678
    • 1

    Description

      The test is setup as follows : 

      • Deploy eventing function with n1ql queries inside
      • Load the source bucket
      • Run some access phase mutations

      We have observed the following error log in eventing.log in N1QL Based Eventing Tests. 

      2021-03-19T16:59:15.178-07:00 [Info] eventing-consumer [worker_perf-test1_23:/tmp/127.0.0.1:8091_23_3374800134.sock:123412] {  "message": "SDK error : LCB_ERR_INDEX_FAILURE (402) Query error : {\n\"requestID\": \"20e0a0f0-bff8-4f46-848e-fdbae6761a0f\",\n\"clientContextID\": \"8@perf-test1.js(OnUpdate)\",\n\"errors\": [{\"code\":12003,\"msg\":\"Keyspace not found in CB datastore: default:eventing-bucket-1 - cause: No bucket named eventing-bucket-1\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"164.504346ms\",\"executionTime\": \"164.433732ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 47,\"errorCount\": 1}\n}\n",  "stack": "Error\n    at OnUpdate (perf-test1.js:8:21)"} Line: 7 Code: var upsert_query = N1QL('UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, \'Hello World\');', {'$docId':docId}, { 'consistency': 'none' }); Stack: Error    at OnUpdate (perf-test1.js:8:21) 

      At this point of time in the test, we have initially begun to load the source bucket- bucket-1 of the eventing function 

      The error log points to another bucket - the destination bucket - eventing-bucket-1 which is created along with the source bucket. The log complains that the bucket is not found but the bucket is not deleted during the test.  The test after this point runs fine and the N1QL queries are executed through eventing. Finally eventing reports this as n1ql_op_exception_count = 1.

      At the same timestamp in query.log we see this 

      2021-03-19T16:59:15.000-07:00 [INFO]  Trying with http://127.0.0.1:8091/pools/default/bucketsStreaming/eventing-bucket-1 
      _time=2021-03-19T17:23:26.495-07:00 _level=INFO _msg=Created New Bucket eventing 

      Looks like query is still trying to connect to the eventing-bucket-1 bucket and in the meanwhile one operation fails. This is not seen in recent 6.6.2 builds : http://perf.jenkins.couchbase.com/job/themis/9165/ 

      The run on 7.0 : http://perf.jenkins.couchbase.com/job/themis/9436/console 

      Logs : 

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.16.zip - KV + N1QL

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.17.zip - KV + N1QL

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.20.zip - KV + N1QL

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.23.zip - KV + N1QL

      https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.97.177.zip - Eventing

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) created issue -
          prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
          Field Original Value New Value
          Description The test is setup as follows : 
           * Deploy eventing function with n1ql queries inside
           * Load the source bucket
           * Run some access phase mutations

          We have observed the following error log in eventing.log in N1QL Based Eventing Tests. 
          {noformat}
          2021-03-19T16:59:15.178-07:00 [Info] eventing-consumer [worker_perf-test1_23:/tmp/127.0.0.1:8091_23_3374800134.sock:123412] { "message": "SDK error : LCB_ERR_INDEX_FAILURE (402) Query error : {\n\"requestID\": \"20e0a0f0-bff8-4f46-848e-fdbae6761a0f\",\n\"clientContextID\": \"8@perf-test1.js(OnUpdate)\",\n\"errors\": [{\"code\":12003,\"msg\":\"Keyspace not found in CB datastore: default:eventing-bucket-1 - cause: No bucket named eventing-bucket-1\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"164.504346ms\",\"executionTime\": \"164.433732ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 47,\"errorCount\": 1}\n}\n", "stack": "Error\n at OnUpdate (perf-test1.js:8:21)"} Line: 7 Code: var upsert_query = N1QL('UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, \'Hello World\');', {'$docId':docId}, { 'consistency': 'none' }); Stack: Error at OnUpdate (perf-test1.js:8:21) {noformat}
          At this point of time in the test, we have initially begun to load the source bucket- bucket-1 of the eventing function 

          The error log points to another bucket - the destination bucket - +*eventing-bucket-1*+ which is created along with the source bucket. The log complains that the bucket is not found but the bucket is not deleted during the test.  The test after this point runs fine and the N1QL queries are executed through eventing. Finally eventing reports this as n1ql_op_exception_count = 1.

          At the same timestamp in query.log we see this 
          {noformat}
          2021-03-19T16:59:15.000-07:00 [INFO] Trying with http://127.0.0.1:8091/pools/default/bucketsStreaming/eventing-bucket-1
          _time=2021-03-19T17:23:26.495-07:00 _level=INFO _msg=Created New Bucket eventing {noformat}
          Looks like query is still trying to connect to the eventing-bucket-1 bucket and in the meanwhile one operation fails. 

          The run : [http://perf.jenkins.couchbase.com/job/themis/9436/console

          Logs : 

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.16.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.17.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.20.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.23.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.97.177.zip] - Eventing
          The test is setup as follows : 
           * Deploy eventing function with n1ql queries inside
           * Load the source bucket
           * Run some access phase mutations

          We have observed the following error log in eventing.log in N1QL Based Eventing Tests. 
          {noformat}2021-03-19T16:59:15.178-07:00 [Info] eventing-consumer [worker_perf-test1_23:/tmp/127.0.0.1:8091_23_3374800134.sock:123412] { "message": "SDK error : LCB_ERR_INDEX_FAILURE (402) Query error : {\n\"requestID\": \"20e0a0f0-bff8-4f46-848e-fdbae6761a0f\",\n\"clientContextID\": \"8@perf-test1.js(OnUpdate)\",\n\"errors\": [{\"code\":12003,\"msg\":\"Keyspace not found in CB datastore: default:eventing-bucket-1 - cause: No bucket named eventing-bucket-1\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"164.504346ms\",\"executionTime\": \"164.433732ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 47,\"errorCount\": 1}\n}\n", "stack": "Error\n at OnUpdate (perf-test1.js:8:21)"} Line: 7 Code: var upsert_query = N1QL('UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, \'Hello World\');', {'$docId':docId}, { 'consistency': 'none' }); Stack: Error at OnUpdate (perf-test1.js:8:21) {noformat}
          At this point of time in the test, we have initially begun to load the source bucket- +*bucket-1*+ of the eventing function 

          The error log points to another bucket - the destination bucket - +*eventing-bucket-1*+ which is created along with the source bucket. The log complains that the bucket is not found but the bucket is not deleted during the test.  The test after this point runs fine and the N1QL queries are executed through eventing. Finally eventing reports this as n1ql_op_exception_count = 1.

          At the same timestamp in query.log we see this 
          {noformat}2021-03-19T16:59:15.000-07:00 [INFO] Trying with http://127.0.0.1:8091/pools/default/bucketsStreaming/eventing-bucket-1
          _time=2021-03-19T17:23:26.495-07:00 _level=INFO _msg=Created New Bucket eventing {noformat}
          Looks like query is still trying to connect to the eventing-bucket-1 bucket and in the meanwhile one operation fails. This is not seen in recent 6.6.2 builds : [http://perf.jenkins.couchbase.com/job/themis/9165/

          The run : [http://perf.jenkins.couchbase.com/job/themis/9436/console

          Logs : 

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.16.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.17.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.20.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.23.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.97.177.zip] - Eventing
          prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
          Description The test is setup as follows : 
           * Deploy eventing function with n1ql queries inside
           * Load the source bucket
           * Run some access phase mutations

          We have observed the following error log in eventing.log in N1QL Based Eventing Tests. 
          {noformat}2021-03-19T16:59:15.178-07:00 [Info] eventing-consumer [worker_perf-test1_23:/tmp/127.0.0.1:8091_23_3374800134.sock:123412] { "message": "SDK error : LCB_ERR_INDEX_FAILURE (402) Query error : {\n\"requestID\": \"20e0a0f0-bff8-4f46-848e-fdbae6761a0f\",\n\"clientContextID\": \"8@perf-test1.js(OnUpdate)\",\n\"errors\": [{\"code\":12003,\"msg\":\"Keyspace not found in CB datastore: default:eventing-bucket-1 - cause: No bucket named eventing-bucket-1\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"164.504346ms\",\"executionTime\": \"164.433732ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 47,\"errorCount\": 1}\n}\n", "stack": "Error\n at OnUpdate (perf-test1.js:8:21)"} Line: 7 Code: var upsert_query = N1QL('UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, \'Hello World\');', {'$docId':docId}, { 'consistency': 'none' }); Stack: Error at OnUpdate (perf-test1.js:8:21) {noformat}
          At this point of time in the test, we have initially begun to load the source bucket- +*bucket-1*+ of the eventing function 

          The error log points to another bucket - the destination bucket - +*eventing-bucket-1*+ which is created along with the source bucket. The log complains that the bucket is not found but the bucket is not deleted during the test.  The test after this point runs fine and the N1QL queries are executed through eventing. Finally eventing reports this as n1ql_op_exception_count = 1.

          At the same timestamp in query.log we see this 
          {noformat}2021-03-19T16:59:15.000-07:00 [INFO] Trying with http://127.0.0.1:8091/pools/default/bucketsStreaming/eventing-bucket-1
          _time=2021-03-19T17:23:26.495-07:00 _level=INFO _msg=Created New Bucket eventing {noformat}
          Looks like query is still trying to connect to the eventing-bucket-1 bucket and in the meanwhile one operation fails. This is not seen in recent 6.6.2 builds : [http://perf.jenkins.couchbase.com/job/themis/9165/

          The run : [http://perf.jenkins.couchbase.com/job/themis/9436/console

          Logs : 

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.16.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.17.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.20.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.23.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.97.177.zip] - Eventing
          The test is setup as follows : 
           * Deploy eventing function with n1ql queries inside
           * Load the source bucket
           * Run some access phase mutations

          We have observed the following error log in eventing.log in N1QL Based Eventing Tests. 
          {noformat}2021-03-19T16:59:15.178-07:00 [Info] eventing-consumer [worker_perf-test1_23:/tmp/127.0.0.1:8091_23_3374800134.sock:123412] { "message": "SDK error : LCB_ERR_INDEX_FAILURE (402) Query error : {\n\"requestID\": \"20e0a0f0-bff8-4f46-848e-fdbae6761a0f\",\n\"clientContextID\": \"8@perf-test1.js(OnUpdate)\",\n\"errors\": [{\"code\":12003,\"msg\":\"Keyspace not found in CB datastore: default:eventing-bucket-1 - cause: No bucket named eventing-bucket-1\"}],\n\"status\": \"fatal\",\n\"metrics\": {\"elapsedTime\": \"164.504346ms\",\"executionTime\": \"164.433732ms\",\"resultCount\": 0,\"resultSize\": 0,\"serviceLoad\": 47,\"errorCount\": 1}\n}\n", "stack": "Error\n at OnUpdate (perf-test1.js:8:21)"} Line: 7 Code: var upsert_query = N1QL('UPSERT INTO `eventing-bucket-1` (KEY, VALUE) VALUES ($docId, \'Hello World\');', {'$docId':docId}, { 'consistency': 'none' }); Stack: Error at OnUpdate (perf-test1.js:8:21) {noformat}
          At this point of time in the test, we have initially begun to load the source bucket- +*bucket-1*+ of the eventing function 

          The error log points to another bucket - the destination bucket - +*eventing-bucket-1*+ which is created along with the source bucket. The log complains that the bucket is not found but the bucket is not deleted during the test.  The test after this point runs fine and the N1QL queries are executed through eventing. Finally eventing reports this as n1ql_op_exception_count = 1.

          At the same timestamp in query.log we see this 
          {noformat}2021-03-19T16:59:15.000-07:00 [INFO] Trying with http://127.0.0.1:8091/pools/default/bucketsStreaming/eventing-bucket-1
          _time=2021-03-19T17:23:26.495-07:00 _level=INFO _msg=Created New Bucket eventing {noformat}
          Looks like query is still trying to connect to the eventing-bucket-1 bucket and in the meanwhile one operation fails. This is not seen in recent 6.6.2 builds : [http://perf.jenkins.couchbase.com/job/themis/9165/

          The run on 7.0 : [http://perf.jenkins.couchbase.com/job/themis/9436/console

          Logs : 

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.16.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.17.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.20.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.96.23.zip] - KV + N1QL

          [https://s3-us-west-2.amazonaws.com/perf-artifacts/jenkins-themis-9436/172.23.97.177.zip] - Eventing
          kamini.jagtiani Kamini Jagtiani made changes -
          Assignee Kamini Jagtiani [ kamini.jagtiani ] Isha Kandaswamy [ isha ]
          isha Isha Kandaswamy made changes -
          Component/s test-execution [ 10231 ]
          Component/s query [ 10718 ]
          isha Isha Kandaswamy made changes -
          Assignee Isha Kandaswamy [ isha ] Prajwal‌ Kiran Kumar‌ [ prajwal.kirankumar ]
          prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
          Issue Type Bug [ 1 ] Task [ 3 ]
          prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive) made changes -
          Resolution Cannot Reproduce [ 5 ]
          Status Open [ 1 ] Closed [ 6 ]
          lynn.straus Lynn Straus made changes -
          Fix Version/s 7.0.0 [ 17233 ]
          lynn.straus Lynn Straus made changes -
          Fix Version/s Cheshire-Cat [ 15915 ]

          People

            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
            prajwal.kirankumar Prajwal‌ Kiran Kumar‌ (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty