Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-44485

[System Test][ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan and directory

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Build: 7.0.0-4502
      Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
      Test Cycle: 1

      In the test,

      • there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
      • For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.

      Cluster config:
      fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]
      kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]
      n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]

      Seeing below ERROR messages while adding fts node to the cluster (which is taking unusually long time).

      172.23.97.217 : fts
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:42.709-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:44.198-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:48.764-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:50.801-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:55.663-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:58.615-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:00.035-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:05.089-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      /opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:05.109-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
      

      Test step:

      [2021-02-19T13:43:46-08:00, vijayviji/sshpass:965341] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.97.217 kill -SIGKILL $(pgrep cbft)
      [pull] sequoiatools/cmd
      [2021-02-19T13:43:51-08:00, sequoiatools/cmd:cc4817] 900
      [pull] sequoiatools/couchbase-cli:7.0
      [2021-02-19T13:59:17-08:00, sequoiatools/couchbase-cli:7.0:350950] server-add -c 172.23.97.215:8091 --server-add https://172.23.107.4 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
      [pull] sequoiatools/couchbase-cli:7.0
      [2021-02-19T13:59:33-08:00, sequoiatools/couchbase-cli:7.0:c57f6c] rebalance -c 172.23.97.215:8091 -u Administrator -p password
      

      Logs:

      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.2.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.3.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.4.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.5.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.215.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.216.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.217.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.227.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.232.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.235.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.236.zip
      url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.237.zip

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Build couchbase-server-6.6.2-9587 contains cbft commit 7573e16 with commit message:
            Revert "MB-44485: Upgrade GOVERSION to 1.13.7"

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9587 contains cbft commit 7573e16 with commit message: Revert " MB-44485 : Upgrade GOVERSION to 1.13.7"

            Build couchbase-server-6.6.2-9587 contains cbgt commit 9854fd3 with commit message:
            [BP] MB-44485 - getLeanPlan, hash mismatch between plan and directory

            build-team Couchbase Build Team added a comment - Build couchbase-server-6.6.2-9587 contains cbgt commit 9854fd3 with commit message: [BP] MB-44485 - getLeanPlan, hash mismatch between plan and directory

            Sreekanth Sivasankaran I reproduced this issue on 6.6.2-9580 with the above given steps, but seeing a different error during rebalance failure -

            Rebalance exited with reason {service_rebalance_failed,fts,
            {worker_died,
            {'EXIT',<0.3810.4>,
            {rebalance_failed,
            {service_error,
            <<"rebalance: waitAssignPIndexDone errors: 1, []string{\"no index definition found\"}">>}}}}}.
            Rebalance Operation Id = 5235939fbfedae7632a99424613a28fb
            

            Validated the fix on 6.6.2-9587 multiple times, and not seeing any rebalance failures.

            System test and functional regression are running.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Sreekanth Sivasankaran I reproduced this issue on 6.6.2-9580 with the above given steps, but seeing a different error during rebalance failure - Rebalance exited with reason {service_rebalance_failed,fts, {worker_died, {'EXIT',<0.3810.4>, {rebalance_failed, {service_error, <<"rebalance: waitAssignPIndexDone errors: 1, []string{\"no index definition found\"}">>}}}}}. Rebalance Operation Id = 5235939fbfedae7632a99424613a28fb Validated the fix on 6.6.2-9587 multiple times, and not seeing any rebalance failures. System test and functional regression are running.

            Verified on 6.6.2-9587. Issue is not seen in the functional regression or system test as well. Will close this ticket after completing validation on 7.0 as well as there are 2 releases in the fix versions.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Verified on 6.6.2-9587. Issue is not seen in the functional regression or system test as well. Will close this ticket after completing validation on 7.0 as well as there are 2 releases in the fix versions.

            Regression and system test on 7.0.0-4907 & 7.0.0-4910 did not show this issue, so closing it.

            mihir.kamdar Mihir Kamdar (Inactive) added a comment - Regression and system test on 7.0.0-4907 & 7.0.0-4910 did not show this issue, so closing it.

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              girish.benakappa Girish Benakappa
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty