Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
1
-
Unknown
Description
Build:7.0.0-4584
Test: -test tests/fts/cheshire-cat/test_fts_clusterops_cheshire_cat_coll_crud.yml -scope tests/fts/cheshire-cat/scope_fts_cheshire_cat.yml
Test Cycle: 1
In the test,
- there are 5 buckets, out of which 20 static fts indexes are created on collections of 3 buckets. Mutations are going on these collections
- For the collections on other 2 buckets, we create and drop indexes and no mutations are going on these collections.
- Continuously run queries on the indexes of collections of bucket1 and bucket2
- wait for 15 mins
- kill cbft on 172.23.97.217 and wait for 15 mins
- stop all mutations and wait for 10 mins
- add fts node 172.23.107.4 and start rebalance and wait for 15 mins
- stop create index loop on bucket4 and bucket5
- wait for rebalance to complete
- Once rebalance is complete, kill cbft on 172.23.107.5 and wait for 15 mins
- start mutations on the collections of bucket1, bucket2 and buckt3
- wait for 2 mins and rebalance to remove node 172.23.97.232 and wait for 5 mins
- kill memcached on 172.23.97.237 and wait for 15 mins
- Add data node 172.23.97.232 and rebalance and wait for 10 mins
- kill cbft on 172.23.107.5 ( which was at 2021-02-22T14:33:48-08:00) and wait for 15 mins
- Rebalance out fts node : 172.23.97.217 wait for 15 mins - which failed with
MB-44725 - Rebalance out kv node : 172.23.97.237
We see below panic while the rebalance is going on:
2021-03-02T22:12:30.079-08:00 [WARN] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:627 |
2021-03-02T22:12:30.079-08:00 [WARN] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:627 |
2021-03-02T22:12:30.141-08:00 [WARN] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:627 |
2021-03-02T22:12:30.143-08:00 [WARN] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:627 |
2021-03-02T22:12:30.144-08:00 [WARN] (GOCBCORE) memdClient read failure: EOF -- cbgt.GocbcoreLogger.Log() at gocbcore_utils.go:627 |
panic: close of closed channel
|
|
|
|
|
goroutine 14852831 [running]: |
github.com/couchbase/gocbcore/v9.(*cccpConfigController).Stop(...)
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.1.2/cccpcfgcontroller.go:63 |
github.com/couchbase/gocbcore/v9.(*pollerController).ForceHTTPPoller.func1(0xc025adac80) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.1.2/pollercontroller.go:153 +0x164 |
created by github.com/couchbase/gocbcore/v9.(*pollerController).ForceHTTPPoller
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.1.2/pollercontroller.go:149 +0x3f |
2021-03-02T22:12:31.345-08:00 [INFO] main: /opt/couchbase/bin/cbft started (v0.6.0/5.5.0) |
2021-03-02T22:12:31.345-08:00 [INFO] main: file descriptor limit current: 200000 max: 200000 |
Logs:
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.107.2.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.107.3.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.107.4.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.107.5.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.215.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.216.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.217.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.227.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.232.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.235.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.236.zip
url : https://cb-jira.s3.us-east-2.amazonaws.com/logs/systestmon-1614752640/collectinfo-2021-03-03T062401-ns_1%40172.23.97.237.zip
Attachments
Issue Links
- is caused by
-
-
- Resolved
-