Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-53928

b'{"error":"limting/throttling: the request has been rejected according to regulator

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Resolved
    • Elixir
    • Elixir
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      AMI used: couchbase-serverless-server-7.2.0-2170-x86_64

      This is on Elixir

      • Created a dataplane
      • Created 10 tenants (databases)
      • Created 1 scope and 1 collection in each tenant
      • Loaded 100000 docs in collections of tenant 1
      • Created an fts index on tenant 1

        {"type": "fulltext-index", "name": "idx", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}, "mapping": {"default_mapping": {"properties": {}, "dynamic": false, "enabled": false}, "types": {"db_0_scope_433.db_0_collection_236": {"default_analyzer": "standard", "dynamic": true, "enabled": true}}}, "doc_config": {"mode": "scope.collection.type_field", "type_field": "type"}}, "sourceType": "couchbase", "sourceName": "05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew", "sourceUUID": "", "planParams": {"numReplicas": 1, "indexPartitions": 1}, "sourceParams": {}}
        

      • After 2 secs, tried to get index status. got below error:

      2022-09-30 12:58:04 | ERROR | MainProcess | test_thread | [on_prem_rest_client.urllib_request] b'{"error":"rest_auth: preparePerms, err: index not found","request":"","status":"fail"}\n'
      2022-09-30 12:58:04 | INFO | MainProcess | test_thread | [fts_base.create] Creating fulltext-index idx on i-0951347cee99a4667.sdk.sandbox.nonprod-project-avengers.com
      2022-09-30 12:58:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.create_fts_index] {"type": "fulltext-index", "name": "idx", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}, "mapping": {"default_mapping": {"properties": {}, "dynamic": false, "enabled": false}, "types": {"db_0_scope_433.db_0_collection_236": {"default_analyzer": "standard", "dynamic": true, "enabled": true}}}, "doc_config": {"mode": "scope.collection.type_field", "type_field": "type"}}, "sourceType": "couchbase", "sourceName": "05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew", "sourceUUID": "", "planParams": {"numReplicas": 1, "indexPartitions": 1}, "sourceParams": {}}
      2022-09-30 12:58:04 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=https://i-0951347cee99a4667.sdk.sandbox.nonprod-project-avengers.com:18094/api/index/idx verb=PUT params={"type": "fulltext-index", "name": "idx", "uuid": "", "params": {"store": {"kvStoreName": "mossStore", "mossStoreOptions": {}}, "mapping": {"default_mapping": {"properties": {}, "dynamic": false, "enabled": false}, "types": {"db_0_scope_433.db_0_collection_236": {"default_analyzer": "standard", "dynamic": true, "enabled": true}}}, "doc_config": {"mode": "scope.collection.type_field", "type_field": "type"}}, "sourceType": "couchbase", "sourceName": "05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew", "sourceUUID": "", "planParams": {"numReplicas": 1, "indexPartitions": 1}, "sourceParams": {}} client_cert=None verify=False
      2022-09-30 12:58:05 | INFO | MainProcess | test_thread | [on_prem_rest_client.create_fts_index] Index idx created
      2022-09-30 12:58:07 | INFO | MainProcess | test_thread | [on_prem_rest_client.urllib_request] Making a rest request api=https://i-0951347cee99a4667.sdk.sandbox.nonprod-project-avengers.com:18094/api/index/05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx verb=GET params= client_cert=None verify=False
      2022-09-30 12:58:07 | ERROR | MainProcess | test_thread | [on_prem_rest_client.urllib_request] b'{"error":"limting/throttling: the request has been rejected according to regulator, msg:\\u003cnil\\u003e","request":"","status":"fail"}\n'
      

      Server Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%400xjsn4tsd-onm9fu.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40ciirerzkifzx6khp.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40igaudrojt35xfnr.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      [FTS] https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40l6e4es1yxvj8inen.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      [FTS] https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40nugejikjqbe319c9.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40qe7yu9kfw9cm1h1z.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40srrhispdpgeg0yt.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40ue3ptexrljjmrw6h.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/test/collectinfo-2022-09-30T201538-ns_1%40uyhnng8zolmqgzlf.9hrnwe4r5j4oeelr.sandbox.nonprod-project-avengers.com.zip

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          thejas.orkombu Thejas Orkombu added a comment - - edited

          Girish Benakappa, This looks something similar to what sarthak had filed the other day

          From the logs I see the index create request for idx is successful,

          2022-09-30T19:58:04.954+00:00 [INFO] rest_create_index: create index request received for idx
          ...
          2022-09-30T19:58:05.051+00:00 [INFO] manager_api: index definition created, indexType: fulltext-index, indexName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx, indexUUID: 56a3c556d14e461b
          ...
          2022-09-30T19:58:05.215+00:00 [INFO] janitor: pindexes to add: 1
          2022-09-30T19:58:05.215+00:00 [INFO]   pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99; UUID: 56a3c556d14e461b
          2022-09-30T19:58:05.215+00:00 [INFO] janitor: pindexes to restart: 0
          2022-09-30T19:58:05.217+00:00 [INFO] partition_content_util: rebalance in progress: false
          2022-09-30T19:58:05.220+00:00 [INFO] pindex_bleve: started runBatchWorker: 0 for pindex: /var/cb/data/@fts/05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99.pindex
          2022-09-30T19:58:05.220+00:00 [INFO] pindex_bleve: started runBatchWorker: 1 for pindex: /var/cb/data/@fts/05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99.pindex
          2022-09-30T19:58:05.220+00:00 [INFO] pindex_bleve: started runBatchWorker: 2 for pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99
          2022-09-30T19:58:05.220+00:00 [INFO] pindex_bleve: started runBatchWorker: 3 for pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99
          2022-09-30T19:58:05.220+00:00 [INFO] janitor: feeds to remove: 0
          2022-09-30T19:58:05.220+00:00 [INFO] janitor: feeds to add: 1
          2022-09-30T19:58:05.220+00:00 [INFO]   05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99
          2022-09-30T19:58:05.221+00:00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9.1.8
          ...
          2022-09-30T19:58:05.283+00:00 [INFO] feed_dcp_gocbcore: newGocbcoreDCPFeed, name: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99, indexName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx, server: http://127.0.0.1:8091, bucketName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew, bucketUUID: b3ae144d23d77cff1497e205e89d2f14
          

          and that the throttling and limiting stats for the index idx matches the bucket on which it was created "05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew". another thing is that, that is the only bucket that has throttling and limting going on by the nsstats endpoint output in the logs, so it doesnt look likely that there is a cross-tenant throttling going on over here.
          The following entry is there only for one bucket:

              "regulatorStats:05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew": {
                  "total_RUs_metered": 0,
                  "total_WUs_metered": 116467,
                  "total_read_ops_capped": 0,
                  "total_read_ops_rejected": 0,
                  "total_write_ops_rejected": 1,
                  "total_write_throttle_seconds": 50.75442986600001,
                  "total_read_ops_metering_errs": 0,
                  "total_write_ops_metering_errs": 0,
                  "total_ops_timed_out_while_metering": 0,
                  "total_batch_limiting_timeouts": 0,
                  "total_batch_rejection_backoff_time_ms": 0
              },
          

          Another thing is that, I'm not sure about the exact reason for the rejection of the request happening, my guess is that there is a burst of data thats coming in from the dcp during the initial part of the streaming from the bucket, which affected the immediate next request (I will keep looking into the cause of it by talking to abhinav and michael). However the fail is happening on a GET request from what I gather, which we shouldn't do since its just a index definition fetch. I've already merged the PR for it, its there in 2185 build, so can you please let me know if its still there in 2185?

          thejas.orkombu Thejas Orkombu added a comment - - edited Girish Benakappa , This looks something similar to what sarthak had filed the other day From the logs I see the index create request for idx is successful, 2022 - 09 -30T19: 58 : 04.954 + 00 : 00 [INFO] rest_create_index: create index request received for idx ... 2022 - 09 -30T19: 58 : 05.051 + 00 : 00 [INFO] manager_api: index definition created, indexType: fulltext-index, indexName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx, indexUUID: 56a3c556d14e461b ... 2022 - 09 -30T19: 58 : 05.215 + 00 : 00 [INFO] janitor: pindexes to add: 1 2022 - 09 -30T19: 58 : 05.215 + 00 : 00 [INFO] pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99; UUID: 56a3c556d14e461b 2022 - 09 -30T19: 58 : 05.215 + 00 : 00 [INFO] janitor: pindexes to restart: 0 2022 - 09 -30T19: 58 : 05.217 + 00 : 00 [INFO] partition_content_util: rebalance in progress: false 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 0 for pindex: /var/cb/data/ @fts /05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99.pindex 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 1 for pindex: /var/cb/data/ @fts /05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99.pindex 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 2 for pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 3 for pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] janitor: feeds to remove: 0 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] janitor: feeds to add: 1 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99 2022 - 09 -30T19: 58 : 05.221 + 00 : 00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9. 1.8 ... 2022 - 09 -30T19: 58 : 05.283 + 00 : 00 [INFO] feed_dcp_gocbcore: newGocbcoreDCPFeed, name: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99, indexName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx, server: http: //127.0.0.1:8091, bucketName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew, bucketUUID: b3ae144d23d77cff1497e205e89d2f14 and that the throttling and limiting stats for the index idx matches the bucket on which it was created "05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew". another thing is that, that is the only bucket that has throttling and limting going on by the nsstats endpoint output in the logs, so it doesnt look likely that there is a cross-tenant throttling going on over here. The following entry is there only for one bucket: "regulatorStats:05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew" : { "total_RUs_metered" : 0 , "total_WUs_metered" : 116467 , "total_read_ops_capped" : 0 , "total_read_ops_rejected" : 0 , "total_write_ops_rejected" : 1 , "total_write_throttle_seconds" : 50.75442986600001 , "total_read_ops_metering_errs" : 0 , "total_write_ops_metering_errs" : 0 , "total_ops_timed_out_while_metering" : 0 , "total_batch_limiting_timeouts" : 0 , "total_batch_rejection_backoff_time_ms" : 0 }, Another thing is that, I'm not sure about the exact reason for the rejection of the request happening, my guess is that there is a burst of data thats coming in from the dcp during the initial part of the streaming from the bucket, which affected the immediate next request (I will keep looking into the cause of it by talking to abhinav and michael). However the fail is happening on a GET request from what I gather, which we shouldn't do since its just a index definition fetch. I've already merged the PR for it, its there in 2185 build, so can you please let me know if its still there in 2185?

          Given Thejas Orkombu has put in a fix for the GET request, we think if you see an index request being rejected by the regulator is expected behavior - to limit the number of indexes users of a tenant can introduce into the system.

          abhinav Abhi Dangeti added a comment - Given Thejas Orkombu has put in a fix for the GET request, we think if you see an index request being rejected by the regulator is expected behavior - to limit the number of indexes users of a tenant can introduce into the system.

          my guess is that there is a burst of data thats coming in from the dcp during the initial part of the streaming from the bucket, which affected the immediate next request

          Note that the regulator should not indicate the need to reject unless the DB is 30 minutes over quota (i.e. >= 9,000,000 units deficit).

          michael.blow Michael Blow added a comment - my guess is that there is a burst of data thats coming in from the dcp during the initial part of the streaming from the bucket, which affected the immediate next request Note that the regulator should not indicate the need to reject unless the DB is 30 minutes over quota (i.e. >= 9,000,000 units deficit).

          People

            girish.benakappa Girish Benakappa
            girish.benakappa Girish Benakappa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty