Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
7.1.0
-
Untriaged
-
1
-
Yes
Description
Build : 7.1.0-1738
Operating System: Amazon Linux 2, distribution_version: Amazon Linux release 2 (Karoo) (ARM)
Tests failed in build sanity while getting number of items in the bucket from the API: http://10.0.203.217:8091/pools/default/buckets/default/stats?zoom=minute
Tests:
./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t fts.moving_topology_fts.MovingTopFTS.rebalance_2_nodes_during_index_building,items=30000,cluster=D,F,F,F,index_replicas=1,GROUP=P0,num_partitions=20,fts_quota=700,rebalance_out=True,disableFileTransferRebalance=False
|
|
|
./testrunner -i node_conf.ini -p get-cbcollect-info=True,get-couch-dbinfo=True,skip_cleanup=False,skip_log_scan=False,skip_security_scan=False -t fts.moving_topology_fts.MovingTopFTS.rebalance_2_nodes_during_index_building,items=30000,cluster=D,F,F,F,index_replicas=1,GROUP=P0,num_partitions=20,fts_quota=700,rebalance_out=True,disableFileTransferRebalance=True
|
Errors in test log:
02:24:18 [2021-11-18 10:24:18,045] - [rest_client:2515] INFO - http://10.0.203.217:8091/pools/default/buckets/default/stats?zoom=minute
|
02:24:18 [2021-11-18 10:24:18,182] - [rest_client:2565] ERROR - get_active_key_count() function for bucket default reported an error 'curr_items'
|
02:24:18 [2021-11-18 10:24:18,182] - [rest_client:2566] ERROR - Corresponding bucket stats JSON is {'op': {'samples': {'timestamp': []}, 'samplesCount': 60, 'isPersistent': True, 'lastTStamp': 0, 'interval': 1000}}
|
02:24:20 [2021-11-18 10:24:20,184] - [rest_client:2565] ERROR - get_active_key_count() function for bucket default reported an error 'curr_items'
|
02:24:20 [2021-11-18 10:24:20,185] - [rest_client:2566] ERROR - Corresponding bucket stats JSON is {'op': {'samples': {'timestamp': []}, 'samplesCount': 60, 'isPersistent': True, 'lastTStamp': 0, 'interval': 1000}}
|
02:24:22 [2021-11-18 10:24:22,187] - [rest_client:2565] ERROR - get_active_key_count() function for bucket default reported an error 'curr_items'
|
02:24:22 [2021-11-18 10:24:22,187] - [rest_client:2566] ERROR - Corresponding bucket stats JSON is {'op': {'samples': {'timestamp': []}, 'samplesCount': 60, 'isPersistent': True, 'lastTStamp': 0, 'interval': 1000}}
|
02:24:24 [2021-11-18 10:24:24,190] - [rest_client:2565] ERROR - get_active_key_count() function for bucket default reported an error 'curr_items'
|
02:24:24 [2021-11-18 10:24:24,190] - [rest_client:2566] ERROR - Corresponding bucket stats JSON is {'op': {'samples': {'timestamp': []}, 'samplesCount': 60, 'isPersistent': True, 'lastTStamp': 0, 'interval': 1000}}
|
02:24:26 [2021-11-18 10:24:26,192] - [rest_client:2565] ERROR - get_active_key_count() function for bucket default reported an error 'curr_items'
|
02:24:26 [2021-11-18 10:24:26,192] - [rest_client:2566] ERROR - Corresponding bucket stats JSON is {'op': {'samples': {'timestamp': []}, 'samplesCount': 60, 'isPersistent': True, 'lastTStamp': 0, 'interval': 1000}}
|
Errors seen in Server log:
[stats:error,2021-11-18T10:24:18.180Z,ns_1@10.0.203.217:<0.21460.5>:stats_reader:log_bad_responses:202]Bad replies: [{'ns_1@10.0.203.201',
|
{error,
|
{failed_connect,
|
[{to_address,{"127.0.0.1",9123}},
|
{inet,[inet],econnrefused}]}}}]
|
[ns_server:error,2021-11-18T10:24:20.310Z,ns_1@10.0.203.217:prometheus_cfg<0.413.0>:prometheus_cfg:handle_info:489]Received exit from Prometheus port server - <0.20766.5>: {abnormal,2}. Restarting Prometheus...
|
[ns_server:error,2021-11-18T10:24:20.642Z,ns_1@10.0.203.217:<0.21689.5>:prometheus:post_async:192]Prometheus http request failed:
|
URL: http://127.0.0.1:9123/api/v1/query
|
Body: query=%7Bname%3D~%60kv_curr_items%7Ckv_curr_items_tot%7Ckv_mem_used_bytes%7Ccouch_docs_actual_disk_size%7Ccouch_views_actual_disk_size%7Ckv_ep_db_data_size_bytes%7Ckv_ep_bg_fetched%60%7D+or+kv_vb_curr_items%7Bstate%3D%27replica%27%7D+or+kv_vb_num_non_resident%7Bstate%3D%27active%27%7D+or+label_replace%28sum+by+%28bucket%2C+name%29+%28irate%28kv_ops%7Bop%3D%60get%60%7D%5B1m%5D%29%29%2C+%60name%60%2C%60cmd_get%60%2C+%60%60%2C+%60%60%29+or+label_replace%28irate%28kv_ops%7Bop%3D%60get%60%2Cresult%3D%60hit%60%7D%5B1m%5D%29%2C%60name%60%2C%60get_hits%60%2C%60%60%2C%60%60%29+or+label_replace%28sum+by+%28bucket%29+%28irate%28kv_cmd_lookup%5B1m%5D%29+or+irate%28kv_ops%7Bop%3D~%60set%7Cincr%7Cdecr%7Cdelete%7Cdel_meta%7Cget_meta%7Cset_meta%7Cset_ret_meta%7Cdel_ret_meta%60%7D%5B1m%5D%29%29%2C+%60name%60%2C+%60ops%60%2C+%60%60%2C+%60%60%29+or+sum+by+%28bucket%2C+name%29+%28%7Bname%3D~%60index_data_size%7Cindex_disk_size%7Ccouch_spatial_data_size%7Ccouch_spatial_disk_size%7Ccouch_views_data_size%60%7D%29&timeout=5s
|
Reason: {failed_connect,[{to_address,{"127.0.0.1",9123}},
|
{inet,[inet],econnrefused}]}
|
|
Logs:
test_8.zip