Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-61682

FTS] Error copying index metadata, error backing up index snapshot during rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • 7.6.2
    • 7.6.2
    • fts

    Description

      Build: 7.6.2-3586

      Steps

      1. Create cluster with 5 search nodes 4 Data nodes 2 query node, search service having 16Gb per node
      2. Create 1 bucket with 1 scope and collection
      3. Load 3M sift dataset
      4. Create 1 FTS index with 0 replica and 90 index partitions (idea was 9 not 90 corrected the script)
      5. Run KNN queries with upsert workload for 20 mins
      6. Rebalance in a node

      Seeing the following error during rebalance

      172.23.105.122: fts
      		 
      2024-04-26T05:06:12.749-07:00 [ERRO] rest: error code: 500, msg: rest_pindex_streamer: WriteTo err: error copying index metadata: error backing up index snapshot: segment: /data/@fts/bucket_bucket1_idx_vwlzp-sname-vector_data_35e25a9f04d952f1_142986e4.pindex/store/000000000512.zap copy err: stat /data/@fts/bucket_bucket1_idx_vwlzp-sname-vector_data_35e25a9f04d952f1_142986e4.pindex/store/000000000512.zap: no such file or directory -- rest.ShowErrorBody() at rest.go:59
      		 
       
      		 
       
      		 
      172.23.106.30: fts
      		 
      2024-04-26T05:05:52.350-07:00 [ERRO] rest: error code: 500, msg: rest_pindex_streamer: stream err: lstat /data/@fts/bucket_bucket1_idx_vwlzp-sname-vector_data_35e25a9f04d952f1_1341cb4d.pindex/store/00000000063e.zap: no such file or directory -- rest.ShowErrorBody() at rest.go:59

      Test Logs:

      [2024-04-26T05:05:16-07:00, sequoiatools/couchbase-cli:7.6:5d91d2] server-add -c 172.23.96.198:8091 --server-add https://172.23.106.176 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
      		 
      [pull] sequoiatools/couchbase-cli:7.6
      		 
      [2024-04-26T05:05:30-07:00, sequoiatools/couchbase-cli:7.6:0155ea] rebalance -c 172.23.96.198:8091 -u Administrator -p password
      		 
      [pull] sequoiatools/cmd
      		 
      [2024-04-26T05:05:35-07:00, sequoiatools/cmd:5a3d53] 60
      		 
      [pull] appropriate/curl
      		 
      [2024-04-26T05:06:42-07:00, appropriate/curl:c604ea] -s -u Administrator:password 172.23.96.198:8091/pools/default/rebalanceProgress
      		 
      [pull] sequoiatools/cmd
      		 
      [2024-04-26T05:08:12-07:00, sequoiatools/cmd:3d9c73] 30
      		 
      [pull] sequoiatools/vectorloader
      		 
      [2024-04-26T05:08:49-07:00, sequoiatools/vectorloader:85ce57] -n 172.23.96.198 -u Administrator -p password -b bucket1 -sc scope_0 -coll coll_0 -ds sift -i True -per 1 -update True
      		 
      [pull] sequoiatools/ftsindexmanager

      Link : http://172.23.109.231/job/component_systest_launcher_3/690/console

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. MB-61421 Mitigating operational race-iness between CopyTo & Merge/Persist

          • Critical - Crashes, loss of data, severe memory leak.
          • Open
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarthak.dua Sarthak Dua
              sarthak.dua Sarthak Dua
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty