Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Untriaged
-
Centos 64-bit
-
-
1
-
Unknown
Description
FTS service is going down while deleting of restored fts index.
Build: 7.0.0-4023
Steps to reproduce:
- create the following cluster: kv:fts, kv
- create bucket b1, create scope b1.s1, create collection b1.s1.c1
- create fts index using UI workbench:
{
|
"name": "idx1", |
"type": "fulltext-index", |
"params": { |
"mapping": { |
"types": { |
"s1.c1": { |
"enabled": true, |
"dynamic": true |
}
|
},
|
"default_mapping": { |
"enabled": false, |
"dynamic": true |
},
|
"default_type": "_default", |
"default_analyzer": "standard", |
"default_datetime_parser": "dateTimeOptional", |
"default_field": "_all", |
"store_dynamic": false, |
"index_dynamic": true, |
"docvalues_dynamic": false |
},
|
"store": { |
"indexType": "scorch", |
"kvStoreName": "" |
},
|
"doc_config": { |
"mode": "scope.collection.type_field", |
"type_field": "type", |
"docid_prefix_delim": "", |
"docid_regexp": "" |
}
|
},
|
"sourceType": "couchbase", |
"sourceName": "b1", |
"sourceUUID": "53f5e222ac77c8f177e02555526cd585", |
"sourceParams": {}, |
"planParams": { |
"maxPartitionsPerPIndex": 171, |
"numReplicas": 0, |
"indexPartitions": 6 |
},
|
"uuid": "" |
}
|
- using the following curl call, make a backup:
curl -GET -H "Content-Type: application/json" -u <username>:<password> http://<your_ip>:<your_port>/api/v1/backup |
- Using UI workbench, drop index idx1
- Using the following curl call, restore idx1 from backup:
curl -POST -H "Content-Type: application/json" -u <username>:<password> http://<your_ip>:<your_port>/api/v1/backup -d '@backup.json' |
where backup.json file contains backup record
- delete restored idx1 index using UI workbench
- check logs, you will see something like that:
Service 'fts' exited with status 2. Restarting. Messages: |
2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__f4e0a48a |
2020-12-18T15:44:30.364-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__6ddbfb54 |
panic: close of closed channel
|
|
goroutine 14179 [running]: |
github.com/couchbase/gocbcore/v9.(*cccpConfigController).Stop(0xc001852780) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/cccpcfgcontroller.go:63 +0x2f |
github.com/couchbase/gocbcore/v9.(*pollerController).Stop(0xc0018a9640) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/pollercontroller.go:116 +0x73 |
github.com/couchbase/gocbcore/v9.(*DCPAgent).Close(0xc00011a580, 0x23, 0x1ced920) |
/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/dcpagent.go:323 +0x59 |
created by github.com/couchbase/cbgt.(*gocbcoreDCPAgentMap).closeAgent
|
/home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/feed_dcp_gocbcore.go:243 |
Corresponding logs are attached.
Corresponding babysitter.log fragment:
=========================ERROR REPORT=========================
|
** Generic server <0.279.0> terminating
|
** Last message in was {<0.302.0>,{exit_status,2}}
|
** When Server state == {state,<0.302.0>,
|
{fts,"/opt/couchbase/bin/cbft",
|
["-cfg=metakv",
|
"-uuid=db70ca68d25590580d38269e4c7dce25",
|
"-server=http://127.0.0.1:8091",
|
"-dataDir=/opt/couchbase/var/lib/couchbase/data/@fts",
|
"-tags=feed,janitor,pindex,queryer,cbauth_service",
|
"-auth=cbauth","-extra=172.23.104.247:8091",
|
"-options=startCheckServer=skip,slowQueryLogTimeout=5s,defaultMaxPartitionsPerPIndex=171,bleveMaxResultWindow=10000,failoverAssignAllPrimaries=false,hideUI=true,cbaudit=true,ipv6=false,ftsMemoryQuota=262144000,maxReplicasAllowed=3,bucketTypesAllowed=membase:ephemeral,http2=true,vbuckets=1024",
|
"-bindHttp=172.23.104.247:8094,0.0.0.0:8094",
|
"-bindHttps=:18094",
|
"-tlsCertFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem",
|
"-tlsKeyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem",
|
"-bindGrpc=172.23.104.247:9130,0.0.0.0:9130",
|
"-bindGrpcSsl=172.23.104.247:19130,0.0.0.0:19130"],
|
[via_goport,exit_status,stderr_to_stdout,
|
{env,
|
[{"GOTRACEBACK","single"},
|
{"CBAUTH_REVRPC_URL",
|
"http://%40:986a767737c424f3424ad9755268232c@127.0.0.1:8091/fts"}]}]},
|
{ringbuffer,1190,1024,
|
{[{<<"panic: close of closed channel\n\ngoroutine 14179 [running]:\ngithub.com/couchbase/gocbcore/v9.(*cccpConfigController).Stop(0xc001852780)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/cccpcfgcontroller.go:63 +0x2f\ngithub.com/couchbase/gocbcore/v9.(*pollerController).Stop(0xc0018a9640)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/pollercontroller.go:116 +0x73\ngithub.com/couchbase/gocbcore/v9.(*DCPAgent).Close(0xc00011a580, 0x23, 0x1ced920)\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/dcpagent.go:323 +0x59\ncreated by github.com/couchbase/cbgt.(*gocbcoreDCPAgentMap).closeAgent\n\t/home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/feed_dcp_gocbcore.go:243 +0x163\n">>,
|
938}],
|
[{<<"2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__54820232\n2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__f4e0a48a\n2020-12-18T15:44:30.364-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__6ddbfb54\n">>,
|
252}]}},
|
fts,undefined,[],0}
|
** Reason for termination ==
|
** {abnormal,2}[error_logger:error,2020-12-18T15:44:30.418-08:00,babysitter_of_ns_1@cb.local:<0.279.0>:ale_error_logger_handler:do_log:107]
|
=========================CRASH REPORT=========================
|
crasher:
|
initial call: ns_port_server:init/1
|
pid: <0.279.0>
|
registered_name: []
|
exception exit: {abnormal,2}
|
in function gen_server:handle_common_reply/8 (gen_server.erl, line 751)
|
ancestors: [<0.278.0>,<0.277.0>,ns_child_ports_sup,ns_babysitter_sup,
|
<0.105.0>]
|
message_queue_len: 1
|
messages: [{'EXIT',<0.302.0>,normal}]
|
links: [<0.278.0>]
|
dictionary: []
|
trap_exit: true
|
status: running
|
heap_size: 10958
|
stack_size: 27
|
reductions: 62246
|
neighbours:[ns_server:info,2020-12-18T15:44:30.420-08:00,babysitter_of_ns_1@cb.local:<0.278.0>:supervisor_cushion:handle_info:60]Cushion managed supervisor for fts failed: {abnormal,2}
|
[error_logger:error,2020-12-18T15:44:30.420-08:00,babysitter_of_ns_1@cb.local:<0.278.0>:ale_error_logger_handler:do_log:107]
|
=========================ERROR REPORT=========================
|
** Generic server <0.278.0> terminating
|
** Last message in was {die,{abnormal,2}}
|
** When Server state == {state,fts,5000,-576460509713038526,undefined,
|
infinity}
|
** Reason for termination ==
|
** {abnormal,2}[error_logger:error,2020-12-18T15:44:30.421-08:00,babysitter_of_ns_1@cb.local:<0.278.0>:ale_error_logger_handler:do_log:107]
|
=========================CRASH REPORT=========================
|
crasher:
|
initial call: supervisor_cushion:init/1
|
pid: <0.278.0>
|
registered_name: []
|
exception exit: {abnormal,2}
|
in function gen_server:handle_common_reply/8 (gen_server.erl, line 751)
|
ancestors: [<0.277.0>,ns_child_ports_sup,ns_babysitter_sup,<0.105.0>]
|
message_queue_len: 0
|
messages: []
|
links: [<0.277.0>]
|
dictionary: []
|
trap_exit: true
|
status: running
|
heap_size: 6772
|
stack_size: 27
|
reductions: 8922
|
neighbours:[error_logger:error,2020-12-18T15:44:30.422-08:00,babysitter_of_ns_1@cb.local:<0.277.0>:ale_error_logger_handler:do_log:107]
|
=========================CRASH REPORT=========================
|
crasher:
|
initial call: erlang:apply/2
|
pid: <0.277.0>
|
registered_name: []
|
exception exit: {abnormal,2}
|
in function restartable:loop/4 (src/restartable.erl, line 69)
|
ancestors: [ns_child_ports_sup,ns_babysitter_sup,<0.105.0>]
|
message_queue_len: 0
|
messages: []
|
links: [<0.122.0>]
|
dictionary: []
|
trap_exit: true
|
status: running
|
heap_size: 4185
|
stack_size: 27
|
reductions: 3080
|
neighbours:[error_logger:error,2020-12-18T15:44:30.423-08:00,babysitter_of_ns_1@cb.local:ns_child_ports_sup<0.122.0>:ale_error_logger_handler:do_log:107]
|
=========================SUPERVISOR REPORT=========================
|
supervisor: {local,ns_child_ports_sup}
|
errorContext: child_terminated
|
reason: {abnormal,2}
|
offender: [{pid,<0.277.0>},
|
{id,{fts,"/opt/couchbase/bin/cbft",
|
["-cfg=metakv",
|
"-uuid=db70ca68d25590580d38269e4c7dce25",
|
"-server=http://127.0.0.1:8091",
|
"-dataDir=/opt/couchbase/var/lib/couchbase/data/@fts",
|
"-tags=feed,janitor,pindex,queryer,cbauth_service",
|
"-auth=cbauth","-extra=172.23.104.247:8091",
|
"-options=startCheckServer=skip,slowQueryLogTimeout=5s,defaultMaxPartitionsPerPIndex=171,bleveMaxResultWindow=10000,failoverAssignAllPrimaries=false,hideUI=true,cbaudit=true,ipv6=false,ftsMemoryQuota=262144000,maxReplicasAllowed=3,bucketTypesAllowed=membase:ephemeral,http2=true,vbuckets=1024",
|
"-bindHttp=172.23.104.247:8094,0.0.0.0:8094",
|
"-bindHttps=:18094",
|
"-tlsCertFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem",
|
"-tlsKeyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem",
|
"-bindGrpc=172.23.104.247:9130,0.0.0.0:9130",
|
"-bindGrpcSsl=172.23.104.247:19130,0.0.0.0:19130"],
|
[via_goport,exit_status,stderr_to_stdout,
|
{env,
|
Corresponding debug.log fragment:
2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__f4e0a48a2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__f4e0a48a2020-12-18T15:44:30.364-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__6ddbfb54panic: close of closed channel
|
goroutine 14179 [running]:github.com/couchbase/gocbcore/v9.(*cccpConfigController).Stop(0xc001852780) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/cccpcfgcontroller.go:63 +0x2fgithub.com/couchbase/gocbcore/v9.(*pollerController).Stop(0xc0018a9640) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/pollercontroller.go:116 +0x73github.com/couchbase/gocbcore/v9.(*DCPAgent).Close(0xc00011a580, 0x23, 0x1ced920) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/dcpagent.go:323 +0x59created by github.com/couchbase/cbgt.(*gocbcoreDCPAgentMap).closeAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/feed_dcp_gocbcore.go:243 +0x163
|
Corresponding fts.log fragment:
2020-12-18T15:44:30.364-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__6ddbfb542020-12-18T15:44:30.364-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__6ddbfb54panic: close of closed channel
|
goroutine 14179 [running]:github.com/couchbase/gocbcore/v9.(*cccpConfigController).Stop(0xc001852780) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/cccpcfgcontroller.go:63 +0x2fgithub.com/couchbase/gocbcore/v9.(*pollerController).Stop(0xc0018a9640) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/pollercontroller.go:116 +0x73github.com/couchbase/gocbcore/v9.(*DCPAgent).Close(0xc00011a580, 0x23, 0x1ced920) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/dcpagent.go:323 +0x59created by github.com/couchbase/cbgt.(*gocbcoreDCPAgentMap).closeAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/feed_dcp_gocbcore.go:243 +0x163
|
Corresponding info.log fragment:
2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__f4e0a48a2020-12-18T15:44:30.363-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__f4e0a48a2020-12-18T15:44:30.364-08:00 [INFO] feed_dcp_gocbcore: Close, name: idx1__6ddbfb54panic: close of closed channel
|
goroutine 14179 [running]:github.com/couchbase/gocbcore/v9.(*cccpConfigController).Stop(0xc001852780) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/cccpcfgcontroller.go:63 +0x2fgithub.com/couchbase/gocbcore/v9.(*pollerController).Stop(0xc0018a9640) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/pollercontroller.go:116 +0x73github.com/couchbase/gocbcore/v9.(*DCPAgent).Close(0xc00011a580, 0x23, 0x1ced920) /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/gopkg/go-1.13.7/pkg/mod/github.com/couchbase/gocbcore/v9@v9.0.7/dcpagent.go:323 +0x59created by github.com/couchbase/cbgt.(*gocbcoreDCPAgentMap).closeAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/cbgt/feed_dcp_gocbcore.go:243 +0x163
|