Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44485

[System Test][ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan and directory

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0-4502
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
      Test Cycle: 1

      In the test,

      • there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
      • For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.

      Cluster config:
      fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]
      kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]
      n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]

      Seeing below ERROR messages while adding fts node to the cluster (which is taking unusually long time).

      172.23.97.217 : fts
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:42.709-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:44.198-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:48.764-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:50.801-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:55.663-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:58.615-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:00.035-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:05.089-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      		 
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:05.109-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63

      Test step: 

      [2021-02-19T13:43:46-08:00, vijayviji/sshpass:965341] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.97.217 kill -SIGKILL $(pgrep cbft)
      		 
      [pull] sequoiatools/cmd
      		 
      [2021-02-19T13:43:51-08:00, sequoiatools/cmd:cc4817] 900
      		 
      [pull] sequoiatools/couchbase-cli:7.0
      		 
      [2021-02-19T13:59:17-08:00, sequoiatools/couchbase-cli:7.0:350950] server-add -c 172.23.97.215:8091 --server-add https://172.23.107.4 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
      		 
      [pull] sequoiatools/couchbase-cli:7.0
      		 
      [2021-02-19T13:59:33-08:00, sequoiatools/couchbase-cli:7.0:c57f6c] rebalance -c 172.23.97.215:8091 -u Administrator -p password

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.2.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.4.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.215.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.216.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.217.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.227.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.232.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.235.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.236.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.237.zip

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MB-26775 [FTS] Service rebalance failed: "planner: CfgGetPlanPIndexes err: cfg_metakv_lean: getLeanPlan, hash mismatch between plan contents

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          mentioned in

          Page Loading...

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty