Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-48848

Search: 1K indexes: some indexes' create/delete requests fail with server write time out

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1444
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud_freetier.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat_free_tier.yml

      • Cluster with 3 nodes having kv,n1ql, search, index on all the nodes
      • Create 1 bucket, 100 scopes and 10 collections in each scopes
      • Create 2500 GSI indexes ( 5 on each collection)
      • Load documents on some of the collections
      • Created 1000 indexes: one index (1 partition) on each collection
      • Run queries on each collection
      • Mutate the documents on each collection and wait for all the index to process mutation
      • Run queries on each collection
      • Delete all the indexes

      While delete, we see some of the indexes delete request failing with below error:

       
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:26:41 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:26:41.40542073Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:26:41.405426588Z"}
      {"log":"2021-10-11 04:26:41,405 - ftsindexmanager - INFO - Index bucket_bucket1_idx_acouw_3-country not deleted\n","stream":"stderr","time":"2021-10-11T11:26:41.4055019Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:28:20 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:28:20.786042327Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:28:20.786047725Z"}
      {"log":"2021-10-11 04:28:20,785 - ftsindexmanager - INFO - Index bucket_bucket1_idx_adgon_4-country not deleted\n","stream":"stderr","time":"2021-10-11T11:28:20.786092936Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:30:00 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:30:00.923212599Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:30:00.923231849Z"}
      {"log":"2021-10-11 04:30:00,923 - ftsindexmanager - INFO - Index bucket_bucket1_idx_aduvc_4-country not deleted\n","stream":"stderr","time":"2021-10-11T11:30:00.923294807Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:31:21 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:31:21.934393546Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:31:21.934399013Z"}
      {"log":"2021-10-11 04:31:21,934 - ftsindexmanager - INFO - Index bucket_bucket1_idx_agldw_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:31:21.934477283Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:33:20 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:33:20.398803786Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:33:20.398809214Z"}
      {"log":"2021-10-11 04:33:20,398 - ftsindexmanager - INFO - Index bucket_bucket1_idx_ahtkg_5-country not deleted\n","stream":"stderr","time":"2021-10-11T11:33:20.398867365Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:34:20 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:34:20.403694133Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:34:20.403699798Z"}
      {"log":"2021-10-11 04:34:20,403 - ftsindexmanager - INFO - Index bucket_bucket1_idx_aiewn_3-country not deleted\n","stream":"stderr","time":"2021-10-11T11:34:20.403759432Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:35:44 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:35:44.94422815Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:35:44.944235122Z"}
      {"log":"2021-10-11 04:35:44,944 - ftsindexmanager - INFO - Index bucket_bucket1_idx_aikvt_6-country not deleted\n","stream":"stderr","time":"2021-10-11T11:35:44.944313705Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:38:08 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:38:08.853192131Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:38:08.853200516Z"}
      {"log":"2021-10-11 04:38:08,853 - ftsindexmanager - INFO - Index bucket_bucket1_idx_antxo-country not deleted\n","stream":"stderr","time":"2021-10-11T11:38:08.853275896Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:39:36 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:39:36.600300729Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:39:36.600306424Z"}
      {"log":"2021-10-11 04:39:36,600 - ftsindexmanager - INFO - Index bucket_bucket1_idx_aoimg_1-country not deleted\n","stream":"stderr","time":"2021-10-11T11:39:36.600379561Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:40:36 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:40:36.604656398Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:40:36.604662069Z"}
      {"log":"2021-10-11 04:40:36,604 - ftsindexmanager - INFO - Index bucket_bucket1_idx_aoncw_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:40:36.60472061Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:41:36 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:41:36.609769901Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:41:36.609786473Z"}
      {"log":"2021-10-11 04:41:36,609 - ftsindexmanager - INFO - Index bucket_bucket1_idx_apdar_5-country not deleted\n","stream":"stderr","time":"2021-10-11T11:41:36.609791289Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:43:03 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:43:03.338676459Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:43:03.338681871Z"}
      {"log":"2021-10-11 04:43:03,338 - ftsindexmanager - INFO - Index bucket_bucket1_idx_arpcq_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:43:03.338754226Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:44:43 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:44:43.558393665Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:44:43.55839926Z"}
      {"log":"2021-10-11 04:44:43,558 - ftsindexmanager - INFO - Index bucket_bucket1_idx_arrzp_2-country not deleted\n","stream":"stderr","time":"2021-10-11T11:44:43.558477729Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:46:06 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:46:06.733219398Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:46:06.733225229Z"}
      {"log":"2021-10-11 04:46:06,733 - ftsindexmanager - INFO - Index bucket_bucket1_idx_asgei_8-country not deleted\n","stream":"stderr","time":"2021-10-11T11:46:06.733303406Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:51:15 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:51:15.356630674Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:51:15.356649416Z"}
      {"log":"2021-10-11 04:51:15,356 - ftsindexmanager - INFO - Index bucket_bucket1_idx_avlss_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:51:15.356729203Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:52:43 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:52:43.612149935Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:52:43.612157138Z"}
      {"log":"2021-10-11 04:52:43,612 - ftsindexmanager - INFO - Index bucket_bucket1_idx_aybwi_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:52:43.612236209Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:54:35 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:54:35.61844407Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:54:35.61844998Z"}
      {"log":"2021-10-11 04:54:35,618 - ftsindexmanager - INFO - Index bucket_bucket1_idx_azxzn_6-country not deleted\n","stream":"stderr","time":"2021-10-11T11:54:35.618524803Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:55:52 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:55:52.310552328Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:55:52.310557913Z"}
      {"log":"2021-10-11 04:55:52,310 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bbqou_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:55:52.310628504Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:57:21 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:57:21.332731315Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:57:21.332736696Z"}
      {"log":"2021-10-11 04:57:21,332 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bcrbi_9-country not deleted\n","stream":"stderr","time":"2021-10-11T11:57:21.332793129Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 11:58:21 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T11:58:21.335626219Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T11:58:21.335631878Z"}
      {"log":"2021-10-11 04:58:21,335 - ftsindexmanager - INFO - Index bucket_bucket1_idx_beerm_2-country not deleted\n","stream":"stderr","time":"2021-10-11T11:58:21.335677837Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:00:20 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:00:20.08939729Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:00:20.089402806Z"}
      {"log":"2021-10-11 05:00:20,089 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bfncg_1-country not deleted\n","stream":"stderr","time":"2021-10-11T12:00:20.089500175Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:01:50 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:01:50.842962747Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:01:50.842968183Z"}
      {"log":"2021-10-11 05:01:50,842 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bgwub_2-country not deleted\n","stream":"stderr","time":"2021-10-11T12:01:50.843013302Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:03:45 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:03:45.337220677Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:03:45.337227885Z"}
      {"log":"2021-10-11 05:03:45,337 - ftsindexmanager - INFO - Index bucket_bucket1_idx_biamb_6-country not deleted\n","stream":"stderr","time":"2021-10-11T12:03:45.337318675Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:05:04 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:05:04.875998095Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:05:04.876003616Z"}
      {"log":"2021-10-11 05:05:04,875 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bjhey-country not deleted\n","stream":"stderr","time":"2021-10-11T12:05:04.876082444Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:06:41 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:06:41.797304025Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:06:41.79730954Z"}
      {"log":"2021-10-11 05:06:41,797 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bkodj_9-country not deleted\n","stream":"stderr","time":"2021-10-11T12:06:41.797371716Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:08:06 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:08:06.062809085Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:08:06.062814304Z"}
      {"log":"2021-10-11 05:08:06,062 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bmpee_5-country not deleted\n","stream":"stderr","time":"2021-10-11T12:08:06.062818473Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:12:11 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:12:11.873106369Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:12:11.873116029Z"}
      {"log":"2021-10-11 05:12:11,872 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bprtu_4-country not deleted\n","stream":"stderr","time":"2021-10-11T12:12:11.873201559Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:14:26 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:14:26.315771435Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:14:26.315777137Z"}
      {"log":"2021-10-11 05:14:26,315 - ftsindexmanager - INFO - Index bucket_bucket1_idx_brzmb_4-country not deleted\n","stream":"stderr","time":"2021-10-11T12:14:26.315836471Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:15:48 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:15:48.64178347Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:15:48.64178932Z"}
      {"log":"2021-10-11 05:15:48,641 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bsphi_7-country not deleted\n","stream":"stderr","time":"2021-10-11T12:15:48.64186182Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:21:50 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:21:50.314737675Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:21:50.314743351Z"}
      {"log":"2021-10-11 05:21:50,314 - ftsindexmanager - INFO - Index bucket_bucket1_idx_bzruz_3-country not deleted\n","stream":"stderr","time":"2021-10-11T12:21:50.314855586Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:24:40 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:24:40.356245403Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:24:40.356250906Z"}
      {"log":"2021-10-11 05:24:40,356 - ftsindexmanager - INFO - Index bucket_bucket1_idx_cbzpf_1-country not deleted\n","stream":"stderr","time":"2021-10-11T12:24:40.356313907Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:28:02 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:28:02.403818786Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:28:02.403824311Z"}
      {"log":"2021-10-11 05:28:02,403 - ftsindexmanager - INFO - Index bucket_bucket1_idx_cducm_4-country not deleted\n","stream":"stderr","time":"2021-10-11T12:28:02.403896303Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:29:02 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:29:02.406823322Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:29:02.406828799Z"}
      {"log":"2021-10-11 05:29:02,406 - ftsindexmanager - INFO - Index bucket_bucket1_idx_ceeia_2-country not deleted\n","stream":"stderr","time":"2021-10-11T12:29:02.406889138Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:30:08 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:30:08.348209444Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:30:08.348215082Z"}
      {"log":"2021-10-11 05:30:08,348 - ftsindexmanager - INFO - Index bucket_bucket1_idx_cgxmh_5-country not deleted\n","stream":"stderr","time":"2021-10-11T12:30:08.348280961Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 12:59:22 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T12:59:22.003570269Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T12:59:22.003575786Z"}
      {"log":"2021-10-11 05:59:22,003 - ftsindexmanager - INFO - Index bucket_bucket1_idx_duplt_1-country not deleted\n","stream":"stderr","time":"2021-10-11T12:59:22.003620846Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:02:17 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:02:17.752395353Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:02:17.752401078Z"}
      {"log":"2021-10-11 06:02:17,752 - ftsindexmanager - INFO - Index bucket_bucket1_idx_dxmqi_8-country not deleted\n","stream":"stderr","time":"2021-10-11T13:02:17.752488862Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:03:47 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:03:47.200498963Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:03:47.200504656Z"}
      {"log":"2021-10-11 06:03:47,200 - ftsindexmanager - INFO - Index bucket_bucket1_idx_dxzqz_4-country not deleted\n","stream":"stderr","time":"2021-10-11T13:03:47.20055368Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:04:55 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:04:55.537885421Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:04:55.537891011Z"}
      {"log":"2021-10-11 06:04:55,537 - ftsindexmanager - INFO - Index bucket_bucket1_idx_dzezw_3-country not deleted\n","stream":"stderr","time":"2021-10-11T13:04:55.537936757Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:06:17 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:06:17.563152125Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:06:17.563196456Z"}
      {"log":"2021-10-11 06:06:17,563 - ftsindexmanager - INFO - Index bucket_bucket1_idx_eculr_3-country not deleted\n","stream":"stderr","time":"2021-10-11T13:06:17.563274491Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:09:06 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:09:06.290488273Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:09:06.290493866Z"}
      {"log":"2021-10-11 06:09:06,290 - ftsindexmanager - INFO - Index bucket_bucket1_idx_egbzg_6-country not deleted\n","stream":"stderr","time":"2021-10-11T13:09:06.290538143Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:10:06 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:10:06.298575449Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:10:06.298581107Z"}
      {"log":"2021-10-11 06:10:06,298 - ftsindexmanager - INFO - Index bucket_bucket1_idx_eghcm_5-country not deleted\n","stream":"stderr","time":"2021-10-11T13:10:06.29863893Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:14:21 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:14:21.281625287Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:14:21.281630697Z"}
      {"log":"2021-10-11 06:14:21,281 - ftsindexmanager - INFO - Index bucket_bucket1_idx_ejqoh_8-country not deleted\n","stream":"stderr","time":"2021-10-11T13:14:21.281699815Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:15:33 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:15:33.261778177Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:15:33.261783645Z"}
      {"log":"2021-10-11 06:15:33,261 - ftsindexmanager - INFO - Index bucket_bucket1_idx_ekjpi_1-country not deleted\n","stream":"stderr","time":"2021-10-11T13:15:33.261855526Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:17:04 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:17:04.583303388Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:17:04.583309048Z"}
      {"log":"2021-10-11 06:17:04,583 - ftsindexmanager - INFO - Index bucket_bucket1_idx_ekxco_1-country not deleted\n","stream":"stderr","time":"2021-10-11T13:17:04.583354656Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:19:12 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:19:12.538566513Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:19:12.538572199Z"}
      {"log":"2021-10-11 06:19:12,538 - ftsindexmanager - INFO - Index bucket_bucket1_idx_emfqu_7-country not deleted\n","stream":"stderr","time":"2021-10-11T13:19:12.538645772Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:21:32 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:21:32.624370867Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:21:32.624376202Z"}
      {"log":"2021-10-11 06:21:32,624 - ftsindexmanager - INFO - Index bucket_bucket1_idx_eofsg_5-country not deleted\n","stream":"stderr","time":"2021-10-11T13:21:32.624452855Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:23:58 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:23:58.766916018Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:23:58.766921451Z"}
      {"log":"2021-10-11 06:23:58,766 - ftsindexmanager - INFO - Index bucket_bucket1_idx_epqyx_3-country not deleted\n","stream":"stderr","time":"2021-10-11T13:23:58.766992156Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:25:06 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:25:06.387497983Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:25:06.38750354Z"}
      {"log":"2021-10-11 06:25:06,387 - ftsindexmanager - INFO - Index bucket_bucket1_idx_erhrm_9-country not deleted\n","stream":"stderr","time":"2021-10-11T13:25:06.387507729Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:31:45 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:31:45.76444473Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:31:45.764450362Z"}
      {"log":"2021-10-11 06:31:45,764 - ftsindexmanager - INFO - Index bucket_bucket1_idx_fhcan_2-country not deleted\n","stream":"stderr","time":"2021-10-11T13:31:45.764525795Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:32:59 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:32:59.783777968Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:32:59.783783715Z"}
      {"log":"2021-10-11 06:32:59,783 - ftsindexmanager - INFO - Index bucket_bucket1_idx_fjbhm_1-country not deleted\n","stream":"stderr","time":"2021-10-11T13:32:59.78385706Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:34:07 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:34:07.981915915Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:34:07.981921676Z"}
      {"log":"2021-10-11 06:34:07,981 - ftsindexmanager - INFO - Index bucket_bucket1_idx_flaqx_3-country not deleted\n","stream":"stderr","time":"2021-10-11T13:34:07.981994621Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:37:16 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:37:16.934202869Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:37:16.93420881Z"}
      {"log":"2021-10-11 06:37:16,934 - ftsindexmanager - INFO - Index bucket_bucket1_idx_fmxtw_5-country not deleted\n","stream":"stderr","time":"2021-10-11T13:37:16.934254978Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:42:42 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:42:42.14749806Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:42:42.147503543Z"}
      {"log":"2021-10-11 06:42:42,147 - ftsindexmanager - INFO - Index bucket_bucket1_idx_ftogy_1-country not deleted\n","stream":"stderr","time":"2021-10-11T13:42:42.147604976Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:45:53 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:45:53.450119072Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:45:53.450126076Z"}
      {"log":"2021-10-11 06:45:53,449 - ftsindexmanager - INFO - Index bucket_bucket1_idx_gblze_5-country not deleted\n","stream":"stderr","time":"2021-10-11T13:45:53.450207957Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:49:01 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:49:01.520105346Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:49:01.520114748Z"}
      {"log":"2021-10-11 06:49:01,519 - ftsindexmanager - INFO - Index bucket_bucket1_idx_gfccs_8-country not deleted\n","stream":"stderr","time":"2021-10-11T13:49:01.520190137Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 13:56:42 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T13:56:42.582537854Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T13:56:42.582543233Z"}
      {"log":"2021-10-11 06:56:42,582 - ftsindexmanager - INFO - Index bucket_bucket1_idx_gmugv_6-country not deleted\n","stream":"stderr","time":"2021-10-11T13:56:42.582594555Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 14:01:02 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T14:01:02.379906105Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T14:01:02.37991249Z"}
      {"log":"2021-10-11 07:01:02,379 - ftsindexmanager - INFO - Index bucket_bucket1_idx_grsli_6-country not deleted\n","stream":"stderr","time":"2021-10-11T14:01:02.379995357Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 14:02:47 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T14:02:47.121075552Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T14:02:47.121081556Z"}
      {"log":"2021-10-11 07:02:47,120 - ftsindexmanager - INFO - Index bucket_bucket1_idx_gsmra_7-country not deleted\n","stream":"stderr","time":"2021-10-11T14:02:47.121159589Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 14:05:30 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T14:05:30.565157617Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T14:05:30.565164721Z"}
      {"log":"2021-10-11 07:05:30,565 - ftsindexmanager - INFO - Index bucket_bucket1_idx_gudwj_4-country not deleted\n","stream":"stderr","time":"2021-10-11T14:05:30.565227223Z"}
      --
      {"log":"Response : {'date': 'Mon, 11 Oct 2021 14:19:04 GMT', 'content-length': '22', 'content-type': 'text/plain; charset=utf-8', 'status': '503'} \n","stream":"stderr","time":"2021-10-11T14:19:04.701193957Z"}
      {"log":"Content : b'server write time out.'\n","stream":"stderr","time":"2021-10-11T14:19:04.701199422Z"}
      {"log":"2021-10-11 07:19:04,701 - ftsindexmanager - INFO - Index bucket_bucket1_idx_hktly_6-country not deleted\n","stream":"stderr","time":"2021-10-11T14:19:04.701266912Z"} 
      

      Logs:

      https://cb-jira.s3.us-east-2.amazonaws.com/logs/delete/collectinfo-2021-10-11T160158-ns_1%40172.31.50.113.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/delete/collectinfo-2021-10-11T160158-ns_1%40172.31.50.61.zip
      https://cb-jira.s3.us-east-2.amazonaws.com/logs/delete/collectinfo-2021-10-11T160158-ns_1%40172.31.50.69.zip

      Logs might have been rotated. but please review.

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty