Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build: 7.0.0-4502
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
In the test,
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
Cluster config:
fts : 5 ===== > [172.23.107.2:8091 172.23.107.3:8091 172.23.107.5:8091 172.23.97.216:8091 172.23.97.217:8091]
kv : 4 ===== > [172.23.97.215:8091 172.23.97.232:8091 172.23.97.235:8091 172.23.97.237:8091]
n1ql : 2 ===== > [172.23.97.227:8091 172.23.97.236:8091]
Seeing below ERROR messages while adding fts node to the cluster (which is taking unusually long time).
172.23.97.217 : fts
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:42.709-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:44.198-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:48.764-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:50.801-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:55.663-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:33:58.615-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:00.035-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:05.089-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
/opt/couchbase/var/lib/couchbase/logs/fts.log.1.gz:2021-02-20T01:34:05.109-08:00 [ERRO] rest: error code: 500, msg: could not retrieve defs: cfg_metakv_lean: getLeanPlan, hash mismatch between plan hash: 91ac106035978664a594f5338e3ff467 contents: {"uuid":"3241daf74e3526d8","planPIndexes":{},"implVersion":"5.5.0","warnings":{}}, and directory stamp: 98b7bc090d97ed42c3cae0db1154bc37 -- rest.ShowErrorBody() at rest.go:63
|
Test step:
[2021-02-19T13:43:46-08:00, vijayviji/sshpass:965341] sshpass -p couchbase ssh -o StrictHostKeyChecking=no root@172.23.97.217 kill -SIGKILL $(pgrep cbft)
|
[pull] sequoiatools/cmd
|
[2021-02-19T13:43:51-08:00, sequoiatools/cmd:cc4817] 900
|
[pull] sequoiatools/couchbase-cli:7.0
|
[2021-02-19T13:59:17-08:00, sequoiatools/couchbase-cli:7.0:350950] server-add -c 172.23.97.215:8091 --server-add https://172.23.107.4 -u Administrator -p password --server-add-username Administrator --server-add-password password --services fts
|
[pull] sequoiatools/couchbase-cli:7.0
|
[2021-02-19T13:59:33-08:00, sequoiatools/couchbase-cli:7.0:c57f6c] rebalance -c 172.23.97.215:8091 -u Administrator -p password
|
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1613818917/collectinfo-2021-02-20T110158-ns_1%40172.23.97.237.zip