Details
-
Bug
-
Resolution: Fixed
-
Major
-
Cheshire-Cat
-
Centos 7; Couchbase EE 7.0.0-3874
-
Untriaged
-
Centos 64-bit
-
1
-
Unknown
Description
Summary
Build index ('gsi-223') got stuck. I waited for almost 2 hours after cbcollect but still the status had not changed
Steps to Repro:
1. Create initial nodes
2020-12-03 04:49:07,898 | test | INFO | pool-1-thread-9 | [table_view:display:72] Rebalance Overview
---------------------------++-------------
| Nodes | Services | Status |
---------------------------++-------------
| 172.23.105.175 | kv | Cluster node |
| 172.23.106.233 | ['n1ql'] | <--- IN — |
| 172.23.106.236 | ['n1ql'] | <--- IN — |
| 172.23.106.238 | ['kv,index'] | <--- IN — |
| 172.23.106.250 | ['kv,index'] | <--- IN — |
| 172.23.106.251 | ['kv,index'] | <--- IN — |
---------------------------++-------------
2. Create 1000 collections with 1 item each
---------------------------------------++-----------------------------------------------------------
| Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used |
---------------------------------------++-----------------------------------------------------------
| 4tkxCz8mOx_3oouJ_qJ-23-348000 | couchbase | 2 | none | 0 | 500 | 419430400 | 0 | 565201977 |
| EHVCoriHb-23-220000 | couchbase | 2 | none | 0 | 500 | 419430400 | 0 | 673038089 |
---------------------------------------++-----------------------------------------------------------
3. Create 333 gsi indexes with 2 replicas each
2020-12-03 04:53:05,470 | test | INFO | MainThread | [Collections:create_indexes_and_initialize_queries:145] Creating indexes with defer build
4. Build indexes
2020-12-03 05:39:26,381 | test | INFO | MainThread | [Collections:wait_for_indexes_to_go_online:106] Waiting for indexes to go online
while building 'gsi-223' index, it got stuck.
Observations
from indexer.log (not sure if this is relevant)
2020-12-03T05:39:04.177-08:00 [Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer 172.23.106.250:9100. Error = read tcp 172.23.106.251:54770->172.23.106.250:9100: use of closed network connection. Kill Pipe. |
2020-12-03T05:39:10.682-08:00 [Error] DATP[->dataport ":9103"] Accept() Error: accept tcp4 0.0.0.0:9103: use of closed network connection |
2020-12-03T05:39:10.682-08:00 [Error] receiving packet: read tcp4 172.23.106.251:9103->172.23.105.175:34024: use of closed network connection |
2020-12-03T05:39:10.682-08:00 [Error] DATP[->dataport ":9103"] worker "172.23.105.175:34024" exit: read tcp4 172.23.106.251:9103->172.23.105.175:34024: use of closed network connection |
2020-12-03T05:39:10.699-08:00 [Error] KVSender::sendDelKeyspacesRequest Unexpected Error During Del Buckets Request Projector 172.23.106.238:9999 Topic INIT_STREAM_TOPIC_f61819a2ff21d3d08d62eafa926ba7ac KeyspaceIds [4tkxCz8mOx_3oouJ_qJ-23-348000:--23-367000:P6Dklm-23-367000]. Err genServer.closed |
