Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60662

Service 'fts' exited with status 1 - structure needs cleaning

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.6.0
    • 7.6.0
    • fts
    • Couchbase server
      Enterprise Edition 7.6.0 build 2090

    Description

      1. Created a 6 node cluster. (kv, kv:fts, fts, kv:fts, kv:fts, fts). Total FTS quota: 12GB

      2. Created 2 buckets standard_bucket_1 and standard_bucket_2 and loaded 50000 docs in each bucket(non vector data).

      3. Created 2 more buckets vector-bucket-0 and vector-bucket-1 and loaded sift 1M. Dataset into each bucket.

      4. Created 2 indexes each on standard_bucket_1 and standard_bucket_2.

      5. Created one index each on vector-bucket-0 and vector-bucket-1. The index was created on the 'vector_data' field.

      6. While the indexing was still in progress, added a new node with KV,FTS and started rebalance operation.

      7. Rebalance completed succesfully.

      8. Some time later I see the following crashes

      Service 'fts' exited with status 1. Restarting. Messages:
      		 
      2024-02-04T21:46:49.229-08:00 [INFO] app_herder: indexing over indexQuota: 1572864000, memUsed: 1889489668, preIndexingMemory: 789283, indexes: 1, waiting: 1
      		 
      2024-02-04T21:46:49.236-08:00 [INFO] app_herder: indexing over indexQuota: 1572864000, memUsed: 1889491185, preIndexingMemory: 790800, indexes: 1, waiting: 2
      		 
      2024-02-04T21:46:49.288-08:00 [INFO] app_herder: indexing over indexQuota: 1572864000, memUsed: 1889227585, preIndexingMemory: 527200, indexes: 1, waiting: 3
      		 
      2024-02-04T21:46:51.226-08:00 [INFO] app_herder: indexing over indexQuota: 1572864000, memUsed: 1845457729, preIndexingMemory: 527200, indexes: 1, waiting: 3
      		 
      2024-02-04T21:46:51.773-08:00 [FATA] scorch AsyncError, path: /opt/couchbase/var/lib/couchbase/data/@fts/vector_vector-bucket-0_index_0_7180da522c85d7f7_4c1c5584.pindex/store, treating this as fatal, err: merging err: merging failed: sync /opt/couchbase/var/lib/couchbase/data/@fts/vector_vector-bucket-0_index_0_7180da522c85d7f7_4c1c5584.pindex/store/0000000011d1.zap: structure needs cleaning, stack dump: /opt/couchbase/var/lib/couchbase/data/@fts/dumps/1707112011.fts.stack.dump.txt -- main.initBleveOptions.func2() at init_bleve.go:113

      9. Even though I see these crashes, the indexing eventually completes.

      Testrunner test:

      ./testrunner -i /data/workspace/debian-p0-fts-vset00-00-moving-topology-scorch_vector_sift_7.6_P0/testexec.15523.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=False,stop-on-failure=False,GROUP=P0,index_type=scorch,fts_quota=2048,disableFileTransferRebalance=True,cluster=D:D+F:D+F:D:F:F,vector_dataset=sift,bucket_quota=1024,sirius_url=http://172.23.120.103:4000 -t fts.vector_moving_topology_fts.VectorSearchMovingTopFTS.rebalance_in_during_index_building,items=50000,standard_buckets=2,index_per_bucket=2,num_vector_buckets=2,index_per_vector_bucket=1,default_bucket=False,GROUP=P0;MULTIPLE_CA

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-60899 Vector-Index indexing increases and falls back to 0 in a loop.

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              sarthak.dua Sarthak Dua
              mohsin.ahmed Mohsin Ahmed
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty