Streaming APIs do not use cluster timeout values from ClusterTimeoutOptions
Description
Environment
None
Gerrit Reviews
None
Release Notes Description
None
Attachments
2
- 18 Sep 2023, 05:26 PM
- 18 Sep 2023, 05:01 PM
is duplicated by
Activity
Jared Casey September 20, 2023 at 3:31 PM
Streaming APIs updated to use cluster timeout (when applicable).
Jared Casey September 18, 2023 at 5:29 PM
Output from after changes (showing new timeout tests pass). The snippets are taken from the full log (
), but show that the streaming operation is using the same timeout from the cluster timeout (output is coming from the C++ client) which is set to 1ms.
Analytics
[2023-09-18 12:21:27.606] [60970,673737] [debug] 2ms, open cluster, id: "de4b9e-45d2-bb42-9d99-85d053268a0449", core version: "1.0.0-dp.8+2.205e88f", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"1ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"4500ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev2+g768e80e5 (python/3.9.7)","view_timeout":"75000ms"}}
[2023-09-18 12:21:27.651] [60970,673737] [debug] 0ms, ANALYTICS: client_context_id="01183f-2568-4346-3106-2c695be0fc0964", "SELECT * FROM `test-dataset` LIMIT 1;"
[2023-09-18 12:21:27.651] [60970,673753] [debug] 0ms, [de4b9e-45d2-bb42-9d99-85d053268a0449/9d2ecf-cbc8-7b4a-facf-124fa76d928936] connecting to 127.0.0.1:8095, timeout=2000ms
[2023-09-18 12:21:27.651] [60970,673737] [trace] 0ms, [de4b9e-45d2-bb42-9d99-85d053268a0449/9d2ecf-cbc8-7b4a-facf-124fa76d928936] HTTP request: analytics, method=POST, path="/query/service", client_context_id="01183f-2568-4346-3106-2c695be0fc0964", timeout=1msQuery
[2023-09-18 12:21:35.686] [60970,673737] [debug] 5ms, open cluster, id: "366a29-1661-a54a-1497-d99a6d1acd1d36", core version: "1.0.0-dp.8+2.205e88f", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"4500ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"1ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev2+g768e80e5 (python/3.9.7)","view_timeout":"75000ms"}}
[2023-09-18 12:21:35.724] [60970,673737] [debug] 0ms, QUERY: client_context_id="f23173-a24d-9647-1d4f-6f0edeebec95e1", prep=false, "SELECT * FROM `default` LIMIT 10;", options={"client_context_id":"f23173-a24d-9647-1d4f-6f0edeebec95e1","metrics":false,"timeout":"1ms"}
[2023-09-18 12:21:35.724] [60970,673737] [trace] 0ms, [366a29-1661-a54a-1497-d99a6d1acd1d36/2c5ccf-1329-144d-61a0-fe14dfaa53977d] HTTP request: query, method=POST, path="/query/service", client_context_id="f23173-a24d-9647-1d4f-6f0edeebec95e1", timeout=1msSearch
[2023-09-18 12:21:41.996] [60970,673737] [debug] 3ms, open cluster, id: "84c965-3219-a645-d2ff-07ca812c190ad6", core version: "1.0.0-dp.8+2.205e88f", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"4500ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"1ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev2+g768e80e5 (python/3.9.7)","view_timeout":"75000ms"}}
[2023-09-18 12:21:42.050] [60970,673737] [debug] 0ms, SEARCH: {"term":"auto"}
[2023-09-18 12:21:42.050] [60970,673737] [trace] 0ms, [84c965-3219-a645-d2ff-07ca812c190ad6/da3bac-9b2d-3e49-fc61-6518739588462f] HTTP request: search, method=POST, path="/api/index/test-search-index/query", client_context_id="09eea8-3367-884d-0910-81ee937e33a962", timeout=1msViews
[2023-09-18 12:21:50.310] [60970,673737] [debug] 3ms, open cluster, id: "a8c440-b1eb-dc4f-7a3d-59c19aea4c51f8", core version: "1.0.0-dp.8+2.205e88f", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"4500ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev2+g768e80e5 (python/3.9.7)","view_timeout":"1ms"}}
[2023-09-18 12:21:50.361] [60970,673737] [trace] 0ms, [a8c440-b1eb-dc4f-7a3d-59c19aea4c51f8/3ebc59-5256-8945-eeab-8dfb15b23fb954] HTTP request: views, method=POST, path="/default/_design/dev_test-ddoc/_view/test-view?limit=10", client_context_id="6abdb9-fdad-3f45-cffa-3523186a82f257", timeout=1msJared Casey September 18, 2023 at 5:05 PM
Output from before changes (although running timeout tests added w/ the changes to show the tests fail). The snippets are taken from the full log (
), but show that the streaming operation is using the default 75000ms even though the cluster timeout (output is coming from the C++ client) is set to 1ms.
Analytics
[2023-09-18 11:51:29.946] [56231,643250] [debug] 2ms, open cluster, id: "b65645-ee8e-f24e-6b42-5d7b0e27b619f5", core version: "1.0.0-dp.8+7.7fc6c6e", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"1ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"1000ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev1+g322c272d (python/3.9.7)","view_timeout":"75000ms"}}
[2023-09-18 11:51:29.988] [56231,643250] [debug] 0ms, ANALYTICS: client_context_id="375735-24d5-b64d-657b-16a2bca438acad", "SELECT * FROM `test-dataset` LIMIT 1;"
[2023-09-18 11:51:29.988] [56231,643250] [trace] 0ms, [b65645-ee8e-f24e-6b42-5d7b0e27b619f5/b390df-3902-4942-b679-b0c046ba4cb1a6] HTTP request: analytics, method=POST, path="/query/service", client_context_id="375735-24d5-b64d-657b-16a2bca438acad", timeout=75000msQuery
[2023-09-18 11:51:38.372] [56231,643250] [debug] 1ms, open cluster, id: "cee6bc-2192-2a4e-104c-5aa9827cb8a70f", core version: "1.0.0-dp.8+7.7fc6c6e", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"1000ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"1ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev1+g322c272d (python/3.9.7)","view_timeout":"75000ms"}}
[2023-09-18 11:51:38.404] [56231,643250] [debug] 0ms, QUERY: client_context_id="6f704f-6eec-9e41-045f-84c098f2e94219", prep=false, "SELECT * FROM `default` LIMIT 10;", options={"client_context_id":"6f704f-6eec-9e41-045f-84c098f2e94219","metrics":false,"timeout":"74500ms"}
[2023-09-18 11:51:38.404] [56231,643250] [trace] 0ms, [cee6bc-2192-2a4e-104c-5aa9827cb8a70f/05e15c-1102-4d49-a3a1-b11547176e37a0] HTTP request: query, method=POST, path="/query/service", client_context_id="6f704f-6eec-9e41-045f-84c098f2e94219", timeout=75000msSearch
[2023-09-18 11:51:51.657] [56231,643250] [debug] 4ms, open cluster, id: "97a3d8-b129-c145-1efc-9fa192a175b18f", core version: "1.0.0-dp.8+7.7fc6c6e", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"1000ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"1ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev1+g322c272d (python/3.9.7)","view_timeout":"75000ms"}}
[2023-09-18 11:51:51.705] [56231,643250] [debug] 0ms, SEARCH: {"term":"auto"}
[2023-09-18 11:51:51.705] [56231,643250] [trace] 0ms, [97a3d8-b129-c145-1efc-9fa192a175b18f/8e5468-7279-8846-4a60-9576fd88a5111c] HTTP request: search, method=POST, path="/api/index/test-search-index/query", client_context_id="01153f-b091-2848-61f8-5dda1ad600aba1", timeout=75000msViews
[2023-09-18 11:51:59.985] [56231,643250] [debug] 1ms, open cluster, id: "07b889-c514-0e4c-aadd-e2f341ab598889", core version: "1.0.0-dp.8+7.7fc6c6e", {"bootstrap_nodes":[{"hostname":"127.0.0.1","port":"11210"}],"options":{"analytics_timeout":"75000ms","bootstrap_timeout":"10000ms","config_idle_redial_timeout":"300000ms","config_poll_floor":"50ms","config_poll_interval":"2500ms","connect_timeout":"2000ms","disable_mozilla_ca_certificates":false,"dns_config":{"nameserver":"2600:1702:2302:780::1","port":53,"timeout":"500ms"},"dump_configuration":false,"enable_clustermap_notification":false,"enable_compression":true,"enable_dns_srv":false,"enable_metrics":true,"enable_mutation_tokens":true,"enable_tcp_keep_alive":true,"enable_tls":false,"enable_tracing":true,"enable_unordered_execution":true,"idle_http_connection_timeout":"1000ms","key_value_durable_timeout":"10000ms","key_value_timeout":"2500ms","management_timeout":"75000ms","max_http_connections":0,"metrics_options":{"emit_interval":"600000ms"},"network":"auto","query_timeout":"75000ms","resolve_timeout":"2000ms","search_timeout":"75000ms","show_queries":false,"tcp_keep_alive_interval":"60000ms","tls_verify":"peer","tracing_options":{"analytics_threshold":"1000ms","key_value_threshold":"500ms","management_threshold":"1000ms","orphaned_emit_interval":"10000ms","orphaned_sample_size":64,"query_threshold":"1000ms","search_threshold":"1000ms","threshold_emit_interval":"10000ms","threshold_sample_size":64,"view_threshold":"1000ms"},"transactions_options":{"cleanup_config":{"cleanup_client_attempts":false,"cleanup_lost_attempts":false,"cleanup_window":"0ms","collections":[]},"durability_level":"none","expiration_time":"0ns","query_config":{"scan_consistency":"not_bounded"}},"trust_certificate":"","use_ip_protocol":"any","user_agent_extra":"pycbc/4.1.8.dev1+g322c272d (python/3.9.7)","view_timeout":"1ms"}}
[2023-09-18 11:52:00.028] [56231,643250] [trace] 0ms, [07b889-c514-0e4c-aadd-e2f341ab598889/bc0855-8b2e-fa4e-70b8-b0e0e08a969953] HTTP request: views, method=POST, path="/default/_design/dev_test-ddoc/_view/test-view?limit=10", client_context_id="ee1241-e9ee-564e-763f-9749cc7cc458eb", timeout=75000msFixed
Pinned fields
Click on the next to a field label to start pinning.
Created September 11, 2023 at 5:32 PM
Updated September 20, 2023 at 3:31 PM
Resolved September 20, 2023 at 3:31 PM
Instabug
The Python SDK is not honoring the streaming API cluster timeouts set from the `ClusterTimeoutOptions`. Although the Python SDK correctly sets the timeout options within the core C++ client when a cluster is created, each streaming (n1ql, analytics, search or views) request sets the timeout value to either the timeout in the request options (`QueryOptions`, `SearchOptions`, etc.) or the default timeout. This means that the core C++ client will not use the cluster timeout value as the request includes a timeout.