Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-49351

FTS:ARM:[ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex,

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.1.0-1562
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml -scale 2

      Seeing below errors on fts nodes while adding fts node:

      Node: 172.31.17.66 : fts

      2021-11-02T01:39:19.675+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_bf622821.pindex, err: pindex: could not load PINDEX_META_FILENAME, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_bf622821.pindex, err: open /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_bf622821.pindex/PINDEX_META: no such file or directory -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.124+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_27707880.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_27707880.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.127+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_bf622821.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_bf622821.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.150+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_eded1b4d.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_eded1b4d.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.209+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_50d4748d.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_50d4748d.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990

      Node: 172.31.22.132 : fts

      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-11-02T01:39:19.663+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_82606038.pindex, err: pindex: could not load PINDEX_META_FILENAME, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_82606038.pindex, err: open /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_82606038.pindex/PINDEX_META: no such file or directory -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990

      Node: 172.31.29.243 : fts

      021-11-02T01:39:19.665+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_f00d334a.pindex, err: pindex: could not load PINDEX_META_FILENAME, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_f00d334a.pindex, err: open /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_f00d334a.pindex/PINDEX_META: no such file or directory -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.282+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_3ca06543.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_3ca06543.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.301+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_50bae1f5.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_50bae1f5.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.313+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_d603a756.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_d603a756.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.364+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_f00d334a.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket4_idx_sfqvg_9-Overall_70c1a4ae37ba9e09_f00d334a.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.371+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_87072782.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_87072782.pindex, err: cannot open index, metadata missing -- cbgt.(*Manager).startPIndex() at manager_janitor.go:990
      		 
      2021-11-02T01:39:22.391+00:00 [ERRO] janitor: startPIndex, OpenPIndex error, cleaning up and trying NewPIndex, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_3c71b5aa.pindex, err: pindex: could not open indexType: fulltext-index, path: /data/@fts/bucket_bucket5_idx_yrify_9-country-free_parking-type-price_6ecd9c8a0616fd2d_3c71b5aa.pindex, err: error parsing mapping JSON: unexpected end of JSON input

      Test step:

      [2021-11-02T01:26:19Z, sequoiatools/couchbase-cli:7.0:da6dc7] server-add -c 172.31.18.170:8091 --server-add https://172.31.26.148 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
      		 
      [2021-11-02T01:26:36Z, sequoiatools/couchbase-cli:7.0:c0ee1a] rebalance -c 172.31.18.170:8091 -u Administrator -p password
      		 
      [2021-11-02T01:26:41Z, sequoiatools/cmd:c0547e] 60
      		 
      [2021-11-02T01:27:47Z, sequoiatools/cmd:31abcb] 900
      		 
      รข<86><92>
      		 
       
      		 
      Error occurred on container - sequoiatools/couchbase-cli:7.0:[rebalance -c 172.31.18.170:8091 -u Administrator -p password]
      		 
       
      		 
      docker logs c0ee1a
      		 
      docker start c0ee1a
      		 
       
      		 
      ^A^@^@^@^@^@^@sWARNING: couchbase-cli version 7.0.2-6611-enterprise does not match couchbase server version 7.1.0-1616-enterprise
      		 
      ^A^@^@^@^@^@^@*Unable to display progress bar on this os
      		 
      ^A^@^@^@^@^@^@JERROR: Rebalance failed. See logs for detailed reason. You can try again.

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.17.156.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.17.66.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.18.170.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.18.76.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.21.95.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.22.132.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.22.32.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.25.241.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.26.148.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.29.243.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1635821021/collectinfo-2021-11-02T024342-ns_1%40172.31.30.96.zip

      Cluster spec:

      ########## Cluster config ##################
      		 
      ######  kv : 4 ===== > [172.31.17.156:8091 172.31.18.170:8091 172.31.18.172:8091 172.31.18.76:8091]  ###########
      		 
      ######  fts : 5 ===== > [172.31.17.66:8091 172.31.22.132:8091 172.31.22.32:8091 172.31.25.241:8091 172.31.29.243:8091]  ###########
      		 
      ######  n1ql : 2 ===== > [172.31.21.95:8091 172.31.30.96:8091]  ###########

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MB-49352 FTS:ARM:[ERRO] Service 'fts' exited with status 1. Restarting.

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Sreekanth Sivasankaran Sreekanth Sivasankaran (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty