Details
-
Bug
-
Resolution: Done
-
Critical
-
Cheshire-Cat
-
Enterprise Edition 7.0.0 build 4214
-
Untriaged
-
-
1
-
Unknown
Description
Build: 7.0.0 build 4214
Scenario:
- 9 Node cluster (5 kv + 2 fts + 2 gsi,n1ql)
- 6 buckets with replica=1
- Load 5K docs into each bucket
- Create 50 GSI and 10 FTS index(fts_partitions=6) on each bucket
- Created and dropped 400 GSI on bucket-0
- Created and dropped 200 GSI on bucket-1
Observation:
Seeing indexer process on node 172.23.104.136 crashes every 5-10 minutes with different error messages
Service 'indexer' exited with status 137. Restarting. Messages:
|
2021-01-15T08:40:55.276+00:00 [Info] TK StreamBegin INIT_STREAM bucket-1 408 254875862124290 0 122
|
2021-01-15T08:40:55.277+00:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.105.155:9999 Topic INIT_STREAM_TOPIC_c3bbff64a283bef8737d4090be200203 bucket-1 InstanceIds [6708855476519055408]
|
2021-01-15T08:40:55.277+00:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.105.211:9999 Topic INIT_STREAM_TOPIC_c3bbff64a283bef8737d4090be200203 bucket-1
|
Instances [indexInstance:<instId:6708855476519055408 state:IndexPending definition:<defnID:6364197747808153916 bucket:bucket-1 scope:_default collection:_default isPrimary:true name:bucket-1_primary_171 using:0 exprType:N1QL secExpressions:<ud>([])</ud> partitionScheme:SINGLE whereExpression:<ud>()</ud> > singlePartn:endpoints:"172.23.104.136:9103" > ]
|
2021-01-15T08:40:55.284+00:00 [Info] DATP[->dataport ":9103"] new connection "172.23.105.211:33836" +3
|
2021-01-15T08:42:50.207+00:00 [Info] ServiceMgr::rebalanceJanitor Running Periodic Cleanup
|
|
|
Service 'indexer' exited with status 137. Restarting. Messages:
|
2021-01-15T09:06:04.923+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 1]
|
2021-01-15T09:06:04.923+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 1]
|
2021-01-15T09:06:09.170+00:00 [Info] Plasma: Adaptive memory quota tuning RSS:1774313472, freePercent:10.375149580759093, currentQuota=483183820
|
2021-01-15T09:06:14.170+00:00 [Info] Plasma: Adaptive memory quota tuning RSS:1773682688, freePercent:10.176469002304113, currentQuota=483183820
|
2021-01-15T09:06:16.813+00:00 [Info] ServiceMgr::rebalanceJanitor Running Periodic Cleanup
|
2021-01-15T09:06:17.432+00:00 [Info] memstats {"Alloc":2027195000, "TotalAlloc":7817339888, "Sys":2309383416, "Lookups":0, "Mallocs":66753198,"Frees":57698805, "HeapAlloc":2027195000, "HeapSys":2199617536, "HeapIdle":165298176, "HeapInuse":2034319360,"HeapReleased":51757056, "HeapObjects":9054393,"GCSys":81049600, "LastGC":1610701472703241001,"PauseTotalNs":29726570, "PauseNs":[1813204, 201280], "NumGC":21}
|
2021-01-15T09:06:19.170+00:00 [Info] Plasma: Adaptive memory quota tuning RSS:1773817856, freePercent:10.166929016282268, currentQuota=483183820
|
|
|
Service 'indexer' exited with status 137. Restarting. Messages:
|
2021-01-15T09:03:55.726+00:00 [Info] TK StreamBegin INIT_STREAM bucket-1 408 254875862124290 0 55
|
2021-01-15T09:03:55.740+00:00 [Info] KVSender::sendMutationTopicRequest Success Projector 172.23.105.155:9999 Topic INIT_STREAM_TOPIC_c3bbff64a283bef8737d4090be200203 bucket-1 InstanceIds [4298428202636728347]
|
2021-01-15T09:03:55.740+00:00 [Info] KVSender::sendMutationTopicRequest Projector 172.23.105.211:9999 Topic INIT_STREAM_TOPIC_c3bbff64a283bef8737d4090be200203 bucket-1
|
Instances [indexInstance:<instId:4298428202636728347 state:IndexPending definition:<defnID:2396559713067897538 bucket:bucket-1 scope:_default collection:_default isPrimary:true name:bucket-1_primary_287 using:0 exprType:N1QL secExpressions:<ud>([])</ud> partitionScheme:SINGLE whereExpression:<ud>()</ud> > singlePartn:endpoints:"172.23.104.136:9103" > ]
|
2021-01-15T09:03:55.744+00:00 [Info] DATP[->dataport ":9103"] new connection "172.23.105.211:36962" +3
|
2021-01-15T09:03:55.751+00:00 [Info] TK StreamBegin INIT_STREAM bucket-1 409 113002366805675 0 5
|
TAF test case:
rebalance_new.rebalance_with_services.ServiceRebalanceTests:
|
test_MB_43291,nodes_init=9,services_init=kv-kv-kv-kv-kv-index:n1ql-index:n1ql-fts-fts,standard_buckets=6,replicas=1,num_items=5000,num_gsi_index=50,num_fts_index=10,gsi_index_create_drop=400,fts_index_create_drop=150,fts_index_partition=6,cluster_action=rebalance_out;rebalance_in,target_nodes=kv,num_nodes_to_run=5
|