Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44712

[system Test] Rebalance failed: assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex missing

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 7.0.0
    • Cheshire-Cat
    • fts
    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0-4584

      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
      Test Cycle: 1

      In the test,

      • there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
      • For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
      • Continuously run queries on the indexes of collections of bucket1 and bucket2
      • wait for 15 mins
      • kill cbft on 172.23.97.217 and wait for 15 mins
      • stop all mutations and wait for 10 mins
      • add fts node 172.23.107.4 and start rebalance and wait for 15 mins

       

      Rebalance failed with below:

      Node: 172.23.97.215

      [ns_server:error,2021-03-02T16:42:39.810-08:00,ns_1@172.23.97.215:service_rebalancer-fts<0.29586.24>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.29632.24>,
      		 
                                     {rebalance_failed,
      		 
                                      {service_error,
      		 
                                       <<"assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex missing, indexDef.Name: bucket_bucket1_idx_flwzs-country-free_parking, pindex: bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2, node: 9a3dcd6b2f5475afe210e3dff6a075c3, state: \"primary\", op: promote, planPIndex: &cbgt.PlanPIndex{Name:\"bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2\", UUID:\"1996c7d3d1ba7b1c\", IndexType:\"fulltext-index\", IndexName:\"bucket_bucket1_idx_flwzs-country-free_parking\", IndexUUID:\"116820e26376e9ab\", IndexParams:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_6.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket1\", SourceUUID:\"26aed6012454cfb3f6fb4c16787a7119\", SourceParams:\"{}\", SourcePartitions:\"728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779\", Nodes:map[string]*cbgt.PlanPIndexNode{\"9fb0278bc883795e26424e7f61c3c899\":(*cbgt.PlanPIndexNode)(0xc0128d7f80), \"c0d85b494365ea8ca151ff7ee70a3c54\":(*cbgt.PlanPIndexNode)(0xc0128d7f90)}}">>}}}
      		 
      [user:error,2021-03-02T16:42:39.814-08:00,ns_1@172.23.97.215:<0.8696.0>:ns_orchestrator:log_rebalance_completion:1407]Rebalance exited with reason {service_rebalance_failed,fts,
      		 
                                    {worker_died,
      		 
                                     {'EXIT',<0.29632.24>,
      		 
                                      {rebalance_failed,
      		 
                                       {service_error,
      		 
                                        <<"assignPIndex: update plan, perhaps a concurrent planner won, err: updatePlanPIndexes: planPIndex missing, indexDef.Name: bucket_bucket1_idx_flwzs-country-free_parking, pindex: bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2, node: 9a3dcd6b2f5475afe210e3dff6a075c3, state: \"primary\", op: promote, planPIndex: &cbgt.PlanPIndex{Name:\"bucket_bucket1_idx_flwzs-country-free_parking_116820e26376e9ab_17739eb2\", UUID:\"1996c7d3d1ba7b1c\", IndexType:\"fulltext-index\", IndexName:\"bucket_bucket1_idx_flwzs-country-free_parking\", IndexUUID:\"116820e26376e9ab\", IndexParams:\"{\\\"doc_config\\\":{\\\"docid_prefix_delim\\\":\\\"\\\",\\\"docid_regexp\\\":\\\"\\\",\\\"mode\\\":\\\"scope.collection.type_field\\\",\\\"type_field\\\":\\\"type\\\"},\\\"mapping\\\":{\\\"analysis\\\":{},\\\"default_analyzer\\\":\\\"keyword\\\",\\\"default_datetime_parser\\\":\\\"dateTimeOptional\\\",\\\"default_field\\\":\\\"_all\\\",\\\"default_mapping\\\":{\\\"dynamic\\\":true,\\\"enabled\\\":false},\\\"default_type\\\":\\\"_default\\\",\\\"docvalues_dynamic\\\":true,\\\"index_dynamic\\\":true,\\\"store_dynamic\\\":false,\\\"type_field\\\":\\\"_type\\\",\\\"types\\\":{\\\"scope_6.coll_1\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"properties\\\":{\\\"country\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"country\\\",\\\"type\\\":\\\"text\\\"}]},\\\"free_parking\\\":{\\\"dynamic\\\":false,\\\"enabled\\\":true,\\\"fields\\\":[{\\\"index\\\":true,\\\"name\\\":\\\"free_parking\\\",\\\"type\\\":\\\"boolean\\\"}]}}}}},\\\"store\\\":{\\\"indexType\\\":\\\"scorch\\\",\\\"segmentVersion\\\":15}}\", SourceType:\"gocbcore\", SourceName:\"bucket1\", SourceUUID:\"26aed6012454cfb3f6fb4c16787a7119\", SourceParams:\"{}\", SourcePartitions:\"728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779\", Nodes:map[string]*cbgt.PlanPIndexNode{\"9fb0278bc883795e26424e7f61c3c899\":(*cbgt.PlanPIndexNode)(0xc0128d7f80), \"c0d85b494365ea8ca151ff7ee70a3c54\":(*cbgt.PlanPIndexNode)(0xc0128d7f90)}}">>}}}}}.
      		 
      Rebalance Operation Id = f460daac5c922252e81232e854c9cf81
      		 
      [ns_server:error,2021-03-02T17:23:26.024-08:00,ns_1@172.23.97.215:<0.27650.42>:diag_handler:handle_diag_eval:785]WARNING: /diag/eval:

       

      Logs:

          url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.2.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.3.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.4.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.107.5.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.215.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.216.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.217.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.227.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.232.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.235.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.236.zip
               url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614734499/collectinfo-2021-03-03T012142-ns_1%40172.23.97.237.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-44725 [System Test]service_rebalance_failed,fts: rebalance_failed,inactivity_timeout

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              girish.benakappa Girish Benakappa
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty