Thejas Orkombu added a comment - 02/Oct/22 10:16 PM - edited Girish Benakappa , This looks something similar to what sarthak had filed the other day From the logs I see the index create request for idx is successful, 2022 - 09 -30T19: 58 : 04.954 + 00 : 00 [INFO] rest_create_index: create index request received for idx ... 2022 - 09 -30T19: 58 : 05.051 + 00 : 00 [INFO] manager_api: index definition created, indexType: fulltext-index, indexName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx, indexUUID: 56a3c556d14e461b ... 2022 - 09 -30T19: 58 : 05.215 + 00 : 00 [INFO] janitor: pindexes to add: 1 2022 - 09 -30T19: 58 : 05.215 + 00 : 00 [INFO] pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99; UUID: 56a3c556d14e461b 2022 - 09 -30T19: 58 : 05.215 + 00 : 00 [INFO] janitor: pindexes to restart: 0 2022 - 09 -30T19: 58 : 05.217 + 00 : 00 [INFO] partition_content_util: rebalance in progress: false 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 0 for pindex: /var/cb/data/ @fts /05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99.pindex 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 1 for pindex: /var/cb/data/ @fts /05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99.pindex 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 2 for pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] pindex_bleve: started runBatchWorker: 3 for pindex: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] janitor: feeds to remove: 0 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] janitor: feeds to add: 1 2022 - 09 -30T19: 58 : 05.220 + 00 : 00 [INFO] 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99 2022 - 09 -30T19: 58 : 05.221 + 00 : 00 [INFO] (GOCBCORE) SDK Version: gocbcore/v9. 1.8 ... 2022 - 09 -30T19: 58 : 05.283 + 00 : 00 [INFO] feed_dcp_gocbcore: newGocbcoreDCPFeed, name: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx_56a3c556d14e461b_acbbef99, indexName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew.db_0_scope_433.idx, server: http: //127.0.0.1:8091, bucketName: 05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew, bucketUUID: b3ae144d23d77cff1497e205e89d2f14 and that the throttling and limiting stats for the index idx matches the bucket on which it was created "05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew". another thing is that, that is the only bucket that has throttling and limting going on by the nsstats endpoint output in the logs, so it doesnt look likely that there is a cross-tenant throttling going on over here. The following entry is there only for one bucket: "regulatorStats:05cf4ba8-968e-497f-93ac-ca75ad01cdb3-5ph1ew" : { "total_RUs_metered" : 0 , "total_WUs_metered" : 116467 , "total_read_ops_capped" : 0 , "total_read_ops_rejected" : 0 , "total_write_ops_rejected" : 1 , "total_write_throttle_seconds" : 50.75442986600001 , "total_read_ops_metering_errs" : 0 , "total_write_ops_metering_errs" : 0 , "total_ops_timed_out_while_metering" : 0 , "total_batch_limiting_timeouts" : 0 , "total_batch_rejection_backoff_time_ms" : 0 }, Another thing is that, I'm not sure about the exact reason for the rejection of the request happening, my guess is that there is a burst of data thats coming in from the dcp during the initial part of the streaming from the bucket, which affected the immediate next request (I will keep looking into the cause of it by talking to abhinav and michael). However the fail is happening on a GET request from what I gather, which we shouldn't do since its just a index definition fetch. I've already merged the PR for it, its there in 2185 build, so can you please let me know if its still there in 2185?

Abhi Dangeti added a comment - 26/Oct/22 12:32 PM Given Thejas Orkombu has put in a fix for the GET request, we think if you see an index request being rejected by the regulator is expected behavior - to limit the number of indexes users of a tenant can introduce into the system.

Michael Blow added a comment - 13/Dec/22 10:33 AM my guess is that there is a burst of data thats coming in from the dcp during the initial part of the streaming from the bucket, which affected the immediate next request Note that the regulator should not indicate the need to reject unless the DB is 30 minutes over quota (i.e. >= 9,000,000 units deficit).