Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60899

Vector-Index indexing increases and falls back to 0 in a loop.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 7.6.0
    • 7.6.0
    • fts

    Description

      Build: 7.6.0-2149

      Test to Reproduce:

      ./testrunner -i /data/workspace/debian-p0-fts-vset00-00-moving-topology-scorch_vector_sift_7.6_P0/testexec.10392.ini -p get-cbcollect-info=False,disable_HTP=True,get-logs=True,stop-on-failure=False,GROUP=P0,index_type=scorch,fts_quota=2048,get-cbcollect-info=True,disableFileTransferRebalance=True,cluster=D:D+F:D+F:D:F:F,vector_dataset=sift,bucket_quota=1024,sirius_url=http://172.23.120.103:4000 -t fts.vector_moving_topology_fts.VectorSearchMovingTopFTS.rebalance_in_during_index_building,items=50000,standard_buckets=2,index_per_bucket=2,num_vector_buckets=2,index_per_vector_bucket=1,default_bucket=False,GROUP=P0;MULTIPLE_CA

      Steps involved in Test:

      1. Create a 6 node cluster. (kv, kv:fts, fts, kv:fts, kv:fts, fts). Total FTS quota: 12GB
      2. Create 2 buckets standard_bucket_1 and standard_bucket_2 and loaded 50000 docs in each bucket(non vector data - emp).
      3. Create 2 more buckets vector-bucket-0 and vector-bucket-1 and loaded sift 1M. Dataset into each bucket.
      4. Create 2 indexes each on standard_bucket_1 and standard_bucket_2.
      5. Create one index each on vector-bucket-0 and vector-bucket-1. The index was created on the 'vector_data' field.
      6. Start a rebalance in operation with indexing going on in background
      7. Wait for indexing complete

      Logs indicating successful rebalance in

      [2024-02-20 22:37:28,770] - [on_prem_rest_client:1694] INFO - adding remote node @172.23.123.208:18091 to this cluster @172.23.107.146:8091
      		 
      [2024-02-20 22:37:38,789] - [on_prem_rest_client:2032] INFO - rebalance progress took 10.02 seconds 
      [2024-02-20 22:37:38,790] - [on_prem_rest_client:2033] INFO - sleep for 10 seconds after rebalance...
      		 
      [2024-02-20 22:37:53,545] - [on_prem_rest_client:2867] INFO - Node 172.23.123.208 not part of cluster inactiveAdded
      		 
      [2024-02-20 22:37:53,661] - [on_prem_rest_client:1926] INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.146,ns_1@172.23.122.132,ns_1@172.23.123.129,ns_1@172.23.123.135,ns_1@172.23.123.157,ns_1@172.23.123.160,ns_1@172.23.123.208', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'}
      		 
      [2024-02-20 22:37:53,771] - [on_prem_rest_client:1931] INFO - rebalance operation started
      		 
      [2024-02-20 22:38:03,783] - [on_prem_rest_client:2095] INFO - rebalance percentage : 6.00 %
      		 
      [2024-02-20 22:38:03,784] - [task:899] INFO - Rebalance - status: running, progress: 6.00%
      		 
      [2024-02-20 22:38:23,827] - [on_prem_rest_client:2095] INFO - rebalance percentage : 14.00 %
      		 
      [2024-02-20 22:38:23,827] - [task:899] INFO - Rebalance - status: running, progress: 14.00%
      		 
      [2024-02-20 22:38:43,867] - [on_prem_rest_client:2095] INFO - rebalance percentage : 25.00 %
      		 
      [2024-02-20 22:38:43,868] - [task:899] INFO - Rebalance - status: running, progress: 25.00%
      		 
      [2024-02-20 22:39:03,938] - [on_prem_rest_client:2095] INFO - rebalance percentage : 42.00 %
      		 
      [2024-02-20 22:39:03,938] - [task:899] INFO - Rebalance - status: running, progress: 42.00%
      		 
      [2024-02-20 22:39:23,988] - [task:899] INFO - Rebalance - status: none, progress: 100.00%
      		 
      [2024-02-20 22:39:24,003] - [task:958] INFO - rebalancing was completed with progress: 100% in 90.23061299324036 sec

       

      The indexing of normal indexes over standard buckets got completed quickly 

      [2024-02-20 22:39:24,637] - [fts_base:5163] INFO - Docs in bucket = 50000, docs in FTS index 'standard_bucket_1_index_1': 50000
      		 
      [2024-02-20 22:39:24,679] - [on_prem_rest_client:1167] INFO - Making a rest request api=http://172.23.123.157:8094/api/index/standard_bucket_1_index_2/count verb=GET params= client_cert=None verify=False
      		 
      [2024-02-20 22:39:24,696] - [on_prem_rest_client:2781] INFO - http://172.23.107.146:8091/pools/default/buckets/standard_bucket_1/stats?zoom=minute
      		 
      [2024-02-20 22:39:24,796] - [fts_base:5163] INFO - Docs in bucket = 50000, docs in FTS index 'standard_bucket_1_index_2': 50000
      		 
      [2024-02-20 22:39:24,840] - [on_prem_rest_client:1167] INFO - Making a rest request api=http://172.23.123.157:8094/api/index/standard_bucket_2_index_1/count verb=GET params= client_cert=None verify=False
      		 
      [2024-02-20 22:39:24,858] - [on_prem_rest_client:2781] INFO - http://172.23.107.146:8091/pools/default/buckets/standard_bucket_2/stats?zoom=minute
      		 
      [2024-02-20 22:39:24,963] - [fts_base:5163] INFO - Docs in bucket = 50000, docs in FTS index 'standard_bucket_2_index_1': 50000
      		 
      [2024-02-20 22:39:25,021] - [on_prem_rest_client:1167] INFO - Making a rest request api=http://172.23.123.135:8094/api/index/standard_bucket_2_index_2/count verb=GET params= client_cert=None verify=False
      		 
      [2024-02-20 22:39:25,038] - [on_prem_rest_client:2781] INFO - http://172.23.107.146:8091/pools/default/buckets/standard_bucket_2/stats?zoom=minute
      		 
      [2024-02-20 22:39:25,142] - [fts_base:5163] INFO - Docs in bucket = 50000, docs in FTS index 'standard_bucket_2_index_2': 50000

       

      Indexing of vector indexes is seen to increase to ~400k and then fall back to 0 in a loop

      [2024-02-20 23:03:53,459] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 474534 

      [2024-02-20 23:04:00,320] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0
      		 
      [2024-02-20 23:05:54,123] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0


      [2024-02-20 23:06:05,942] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 479238
      		 
      [2024-02-20 23:07:02,851] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 475930


      [2024-02-20 23:07:09,651] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0
      		 
      [2024-02-20 23:07:53,004] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0


      [2024-02-20 23:57:52,809] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 475330
      		 
      [2024-02-20 23:57:59,690] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 480651


      [2024-02-20 23:59:29,446] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0
      		 

      [2024-02-21 00:01:42,966] - [fts_base:5163] INFO - Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0
      		 
      [2024-02-21 00:01:42,983] - [fts_base:5213] INFO - FTS index count not matching bucket count even after 20 tries: Docs in bucket = 1000000, docs in FTS index 'vector_vector-bucket-0_index_0': 0

       

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-60662 Service 'fts' exited with status 1 - structure needs cleaning

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              thejas.orkombu Thejas Orkombu
              sarthak.dua Sarthak Dua
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty