Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-60236

[Intermittent] [Windows] Index has gone missing after rebalance

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.6.0
    • 7.6.0
    • secondary-index
    • 7.6.0-1970
    • Untriaged
    • 0
    • Unknown

    Description

      Info for QE -

      Test - 
      ./testrunner -i /tmp/testexec.22464.ini -p gsi_type=plasma,reset_services=True,java_sdk_client=True,GROUP=P0,use_gsi_for_primary=True,get-cbcollect-info=True,get-cbcollect-info=True -t gsi.gsi_file_based_rebalance.FileBasedRebalance.test_gsi_swap_rebalance,nodes_init=4,services_init=kv-index:n1ql-index:n1ql-index:n1ql,create_primary_index=False,json_template=Hotel,default_bucket=false,num_of_docs_per_collection=10000,rebalance_all_at_once=true,replica_repair=true,GROUP=P0,reset_services=True
      

      The test does a swap rebalance with replica repair ( before rebalance, a few replicas are dropped to trigger replica repair).

      From the test logs -

      Validation is performed around 22:15:39. The list of indexes before rebalance -

      Index names before rebalance : ['#primary_03YTJUSCCV', '#primary_03YTJUSCCV (replica 1)', '#primary_58Z3E96DEL', '#primary_58Z3E96DEL (replica 1)', '#primary_JZN1GKC0UV', '#primary_JZN1GKC0UV (replica 1)', '#primary_V41BUHLWC1', '#primary_V41BUHLWC1 (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17array_index_checkin', 'hotel082119db71524661a6c012b47b8f5c17array_index_checkin (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17array_index_overall', 'hotel082119db71524661a6c012b47b8f5c17array_index_overall (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17array_index_rooms', 'hotel082119db71524661a6c012b47b8f5c17array_index_rooms (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17flatten_keys', 'hotel082119db71524661a6c012b47b8f5c17flatten_keys (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_array_count', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_array_count (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_avg_rating', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_avg_rating (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17missing_keys', 'hotel082119db71524661a6c012b47b8f5c17missing_keys (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17partitioned_index', 'hotel082119db71524661a6c012b47b8f5c17partitioned_index (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17price', 'hotel082119db71524661a6c012b47b8f5c17price (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_checkin', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_checkin (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_overall', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_overall (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_rooms', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_rooms (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20flatten_keys', 'hotel534a238fa3da42bab915cd390e8b4a20flatten_keys (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_array_count', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_array_count (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_avg_rating', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_avg_rating (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20missing_keys', 'hotel534a238fa3da42bab915cd390e8b4a20missing_keys (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20partitioned_index', 'hotel534a238fa3da42bab915cd390e8b4a20partitioned_index (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20price', 'hotel534a238fa3da42bab915cd390e8b4a20price (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_checkin', 'hotel628b314810a94c33bed9b608b17b8e54array_index_checkin (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_checkin (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_overall', 'hotel628b314810a94c33bed9b608b17b8e54array_index_overall (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_rooms', 'hotel628b314810a94c33bed9b608b17b8e54array_index_rooms (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54flatten_keys', 'hotel628b314810a94c33bed9b608b17b8e54flatten_keys (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54flatten_keys (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_array_count', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_array_count (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_avg_rating', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_avg_rating (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54missing_keys', 'hotel628b314810a94c33bed9b608b17b8e54missing_keys (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54missing_keys (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54partitioned_index', 'hotel628b314810a94c33bed9b608b17b8e54partitioned_index (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54partitioned_index (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54price', 'hotel628b314810a94c33bed9b608b17b8e54price (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54price (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_checkin', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_checkin (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_checkin (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_overall', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_overall (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_overall (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_rooms', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_rooms (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37eflatten_keys', 'hotel7bcdcf711e514532a3f83581a434c37eflatten_keys (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_array_count', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_array_count (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_array_count (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_avg_rating', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_avg_rating (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37emissing_keys', 'hotel7bcdcf711e514532a3f83581a434c37emissing_keys (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37emissing_keys (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37epartitioned_index', 'hotel7bcdcf711e514532a3f83581a434c37epartitioned_index (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37eprice', 'hotel7bcdcf711e514532a3f83581a434c37eprice (replica 1)']
      

      List of indexes after rebalance -

      ['#primary_03YTJUSCCV', '#primary_03YTJUSCCV (replica 1)', '#primary_58Z3E96DEL', '#primary_58Z3E96DEL (replica 1)', '#primary_JZN1GKC0UV', '#primary_JZN1GKC0UV (replica 1)', '#primary_V41BUHLWC1', '#primary_V41BUHLWC1 (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17array_index_checkin', 'hotel082119db71524661a6c012b47b8f5c17array_index_checkin (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17array_index_overall', 'hotel082119db71524661a6c012b47b8f5c17array_index_overall (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17array_index_rooms', 'hotel082119db71524661a6c012b47b8f5c17array_index_rooms (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17flatten_keys', 'hotel082119db71524661a6c012b47b8f5c17flatten_keys (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_array_count', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_array_count (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_avg_rating', 'hotel082119db71524661a6c012b47b8f5c17free_breakfast_avg_rating (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17missing_keys', 'hotel082119db71524661a6c012b47b8f5c17missing_keys (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17partitioned_index', 'hotel082119db71524661a6c012b47b8f5c17partitioned_index (replica 1)', 'hotel082119db71524661a6c012b47b8f5c17price', 'hotel082119db71524661a6c012b47b8f5c17price (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_checkin', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_checkin (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_overall', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_overall (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_rooms', 'hotel534a238fa3da42bab915cd390e8b4a20array_index_rooms (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20flatten_keys', 'hotel534a238fa3da42bab915cd390e8b4a20flatten_keys (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_array_count', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_array_count (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_avg_rating (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20missing_keys', 'hotel534a238fa3da42bab915cd390e8b4a20missing_keys (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20partitioned_index', 'hotel534a238fa3da42bab915cd390e8b4a20partitioned_index (replica 1)', 'hotel534a238fa3da42bab915cd390e8b4a20price', 'hotel534a238fa3da42bab915cd390e8b4a20price (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_checkin', 'hotel628b314810a94c33bed9b608b17b8e54array_index_checkin (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_checkin (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_overall', 'hotel628b314810a94c33bed9b608b17b8e54array_index_overall (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54array_index_rooms', 'hotel628b314810a94c33bed9b608b17b8e54array_index_rooms (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54flatten_keys', 'hotel628b314810a94c33bed9b608b17b8e54flatten_keys (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54flatten_keys (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_array_count', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_array_count (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_avg_rating', 'hotel628b314810a94c33bed9b608b17b8e54free_breakfast_avg_rating (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54missing_keys', 'hotel628b314810a94c33bed9b608b17b8e54missing_keys (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54missing_keys (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54partitioned_index', 'hotel628b314810a94c33bed9b608b17b8e54partitioned_index (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54partitioned_index (replica 2)', 'hotel628b314810a94c33bed9b608b17b8e54price', 'hotel628b314810a94c33bed9b608b17b8e54price (replica 1)', 'hotel628b314810a94c33bed9b608b17b8e54price (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_checkin', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_checkin (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_checkin (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_overall', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_overall (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_overall (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_rooms', 'hotel7bcdcf711e514532a3f83581a434c37earray_index_rooms (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37eflatten_keys', 'hotel7bcdcf711e514532a3f83581a434c37eflatten_keys (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_array_count', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_array_count (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_array_count (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_avg_rating', 'hotel7bcdcf711e514532a3f83581a434c37efree_breakfast_avg_rating (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37emissing_keys', 'hotel7bcdcf711e514532a3f83581a434c37emissing_keys (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37emissing_keys (replica 2)', 'hotel7bcdcf711e514532a3f83581a434c37epartitioned_index', 'hotel7bcdcf711e514532a3f83581a434c37epartitioned_index (replica 1)', 'hotel7bcdcf711e514532a3f83581a434c37eprice', 'hotel7bcdcf711e514532a3f83581a434c37eprice (replica 1)']
      

      This specific index is missing -

      hotel534a238fa3da42bab915cd390e8b4a20free_breakfast_avg_rating
      

      This same test has passed on the same build for the debian job.

      cbcollect attached.

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            pavan.pb Pavan PB
            pavan.pb Pavan PB
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty