============================================================================== couchbase logs (indexer.log) cbbrowse_logs indexer.log ============================================================================== Initializing write barrier = 8000 2021-08-16T13:47:02.076+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:02.076+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:02.081+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:02.090+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:02.090+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:02.090+00:00 [Info] Setting log level to Info 2021-08-16T13:47:02.091+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:47:02.092+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021/08/16 13:47:02 audit: created new audit service 2021-08-16T13:47:02.100+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:02.100+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:02.100+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:02.100+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:02.100+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:02.109+00:00 [Info] security context initialized 2021-08-16T13:47:02.109+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:02.109+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:02.109+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:02.109+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:02.109+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:02.116+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:02.117+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:02.119+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:02.131+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:02.132+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:02.136+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:02.138+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:02.138+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:02.139+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:02.141+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:02.165+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:02.165+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 62 us 2021-08-16T13:47:02.166+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:02.167+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:02.174+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:02.174+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:02.174+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:02.174+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:02.174+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:02.174+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:02.176+00:00 [Info] ClustMgr:handleSetLocalValue Key IndexerId Value 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:02.177+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:02.177+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:02.179+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused 2021-08-16T13:47:02.179+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:03.185+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM Error from Projector Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused 2021-08-16T13:47:03.185+00:00 [Warn] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused. Retrying (1). 2021-08-16T13:47:08.185+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.187+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:08.187+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:08.187+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:08.187+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.187+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:08.187+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:08.187+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:08.187+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:08.187+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:08.187+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:08.187+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:08.190+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap , elapsed: 2.31542ms 2021-08-16T13:47:08.190+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:08.192+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:08.194+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:08.198+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:08.198+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:08.198+00:00 [Warn] Encountered error while reading persisted stats. Skipping read. Error: open /opt/couchbase/var/lib/couchbase/data/@2i/indexstats/stats: no such file or directory 2021-08-16T13:47:08.198+00:00 [Info] ClustMgr:handleIndexerReady mType: CLUST_MGR_INDEXER_READY indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:08.198+00:00 [Info] SettingsMgr::handleIndexerReady apply pending setting changes 2021-08-16T13:47:08.198+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:47:08.198+00:00 [Error] ServiceMgr::getGlobalTopology Error gathering global topology 127.0.0.1:9102/getIndexMetadata Unable to find given hostport in cbauth database: `127.0.0.1:9102' 2021-08-16T13:47:08.198+00:00 [Error] ServiceMgr::updateNodeList Error Fetching Topology Unable to find given hostport in cbauth database: `127.0.0.1:9102' 2021-08-16T13:47:08.198+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.199+00:00 [Info] ClustMgr:handleSetLocalValue Key IndexerNodeUUID Value 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.199+00:00 [Info] ServiceMgr::listenMoveIndex &{8a95cde683482bdec453edbb8b18eb5d 5 } 2021-08-16T13:47:08.200+00:00 [Info] Indexer::NewIndexer Status Active 2021-08-16T13:47:08.200+00:00 [Info] 127.0.0.1:8091 starting RESTful services 2021-08-16T13:47:08.200+00:00 [Info] Indexer::monitorMemUsage started... 2021-08-16T13:47:08.200+00:00 [Info] memstats {"Alloc":0, "TotalAlloc":0, "Sys":0, "Lookups":0, "Mallocs":0,"Frees":0, "HeapAlloc":0, "HeapSys":0, "HeapIdle":0, "HeapInuse":0,"HeapReleased":0, "HeapObjects":0,"MSpanInuse":0, "MSpanSys": 0, "StackInuse": 0,"GCSys":0, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0} 2021-08-16T13:47:08.200+00:00 [Info] progress stats collection done. 2021-08-16T13:47:08.200+00:00 [Info] send progress stats to clients 2021-08-16T13:47:08.201+00:00 [Info] New settings received: {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false} 2021-08-16T13:47:08.202+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:08.202+00:00 [Info] Setting log level to Info 2021-08-16T13:47:08.204+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:47:08.204+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 0] []} 2021-08-16T13:47:08.205+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:47:08.205+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 0] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:08.206+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 0] 2021-08-16T13:47:08.207+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 0] 2021-08-16T13:47:08.208+00:00 [Info] Indexer::updateStorageMode Try setting storage mode to plasma 2021-08-16T13:47:08.208+00:00 [Info] Indexer: Setting GC percent to 100 2021-08-16T13:47:08.208+00:00 [Info] CompactionManager: Refreshing settings 2021-08-16T13:47:08.208+00:00 [Info] MutationMgr::MaxQueueMemoryQuota 53687091 2021-08-16T13:47:08.208+00:00 [Info] ClustMgr:handleConfigUpdate 2021-08-16T13:47:08.210+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CONFIG_UPDATE elapsed 1.269168ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:08.212+00:00 [Info] updator: updating service map. server group=Group 1, indexerVersion=5 nodeAddr ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:8091 clusterVersion 5 excludeNode storageMode 2 2021-08-16T13:47:09.174+00:00 [Info] Indexer::ReadMemstats Time Taken 127.77µs 2021-08-16T13:47:11.670+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:11.706+00:00 [Info] ServiceMgr::PrepareTopologyChange {b82c752c1c8d25413a3ead033ffba421 [] topology-change-rebalance [{{8a95cde683482bdec453edbb8b18eb5d 5 } recovery-full}] []} 2021-08-16T13:47:11.709+00:00 [Info] Indexer::run:msg_loop: INDEXER_CHECK_DDL_IN_PROGRESS message from wrkrPrioRecvCh channel processing took 1.855µs 2021-08-16T13:47:11.709+00:00 [Info] ServiceMgr::prepareRebalance Found DDL Running [] 2021-08-16T13:47:11.709+00:00 [Info] ServiceMgr::prepareRebalance Init Prepare Phase 2021-08-16T13:47:11.709+00:00 [Info] ClustMgr:handleSetLocalValue Key RebalanceRunning Value 2021-08-16T13:47:11.710+00:00 [Info] ClustMgr:handleRebalanceRunning mType: CLUST_MGR_REBALANCE_RUNNING indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:11.710+00:00 [Info] Indexer::run:msg_loop: CLUST_MGR_SET_LOCAL message from wrkrPrioRecvCh channel processing took 1.322903ms 2021-08-16T13:47:11.710+00:00 [Info] ServiceMgr::PrepareTopologyChange Success. isBalanced true 2021-08-16T13:47:11.710+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 1] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.710+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_REBALANCE_RUNNING elapsed 2.511µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:11.710+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 1] [{[0 0 0 0 0 0 0 0] prepare/b82c752c1c8d25413a3ead033ffba421 task-prepared task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]}]} 2021-08-16T13:47:11.711+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 1] 2021-08-16T13:47:11.711+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 1] 2021-08-16T13:47:11.712+00:00 [Info] ServiceMgr::StartTopologyChange {b82c752c1c8d25413a3ead033ffba421 [] topology-change-rebalance [{{8a95cde683482bdec453edbb8b18eb5d 5 } recovery-full}] []} 2021-08-16T13:47:11.724+00:00 [Info] ClustMgr:handleSetLocalValue Key RebalanceToken Value {"MasterId":"8a95cde683482bdec453edbb8b18eb5d","RebalId":"b82c752c1c8d25413a3ead033ffba421","Source":0,"Error":"","MasterIP":"ec2-3-8-56-217.eu-west-2.compute.amazonaws.com"} 2021-08-16T13:47:11.725+00:00 [Info] ServiceMgr::registerLocalRebalanceToken Registered Rebalance Token In Local Meta &{8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} 2021-08-16T13:47:11.726+00:00 [Info] ServiceMgr::registerRebalanceTokenInMetakv Registered Global Rebalance Token In Metakv &{8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::registerGlobalRebalanceToken ClusterInfo Node List doesn't match Known Nodes in Rebalance Request. Skip Rebalance. cinfo.nodes : [{458753 active http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:8092/ ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:8091 map[] 6313 6313 7.691866112e+09 8.274788352e+09 aarch64-unknown-linux-gnu map[direct:11210 distTCP:21100 distTLS:21150 httpsCAPI:18092 httpsMgmt:18091] healthy 105 7.1.0-1152-enterprise true [backup fts index kv n1ql] 8a95cde683482bdec453edbb8b18eb5d inet}], change : {b82c752c1c8d25413a3ead033ffba421 [] topology-change-rebalance [{{8a95cde683482bdec453edbb8b18eb5d 5 } recovery-full}] []} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::startRebalance skip planner due to skipRebalance flag 2021-08-16T13:47:11.733+00:00 [Info] NewRebalancer nodeId 8a95cde683482bdec453edbb8b18eb5d rebalToken &{8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} master true localaddr 127.0.0.1:9102 runPlanner false runParam &{false []} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::StartTopologyChange returns Error . isBalanced true. 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 2] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 2] [{[0 0 0 0 0 0 0 0] prepare/b82c752c1c8d25413a3ead033ffba421 task-prepared task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]} {[0 0 0 0 0 0 0 0] rebalance/b82c752c1c8d25413a3ead033ffba421 task-rebalance task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]}]} 2021-08-16T13:47:11.734+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:47:11.734+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 3] [{[0 0 0 0 0 0 0 0] prepare/b82c752c1c8d25413a3ead033ffba421 task-prepared task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]} {[0 0 0 0 0 0 0 1] rebalance/b82c752c1c8d25413a3ead033ffba421 task-rebalance task-running true 1 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]}]} 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 3] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 3] 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 3] 2021-08-16T13:47:11.736+00:00 [Info] Rebalancer::doFinish Cleanup 2021-08-16T13:47:11.736+00:00 [Info] ServiceMgr::runCleanupPhase path /indexing/rebalance/RebalanceToken isMaster true 2021-08-16T13:47:11.736+00:00 [Info] Rebalancer::processDropIndexQueue Done Received 2021-08-16T13:47:11.737+00:00 [Info] ServiceMgr::cleanupGlobalRToken Delete Global Rebalance Token {8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} 2021-08-16T13:47:11.740+00:00 [Info] ServiceMgr::cleanupLocalRToken Cleanup 2021-08-16T13:47:11.740+00:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceToken 2021-08-16T13:47:11.741+00:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup 2021-08-16T13:47:11.741+00:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning 2021-08-16T13:47:11.742+00:00 [Info] ServiceMgr::onRebalanceDoneLOCKED Rebalance Done: isBalanced true, isMaster true, forceUnbalanced false, err: 2021-08-16T13:47:11.742+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 4] []} 2021-08-16T13:47:11.742+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 4] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.742+00:00 [Info] DDLServiceMgr: handling rebalance done 2021-08-16T13:47:11.743+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 4] []} 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 4] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.745+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.745+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.748+00:00 [Error] DDLServiceMgr:rebalanceDone(): Failed to initialize metadata provider. Error=DDLServiceMgr:rebalanceDone: Failed to initialize metadata provider. Unknown host=map[8a95cde683482bdec453edbb8b18eb5d:true]. 2021-08-16T13:47:11.752+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:11.822+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:24.874+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:24.874+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:25.179+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:25.450+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:26.590+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:27.778+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_SERVICE_MAP elapsed 4.507159ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:27.778+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CLIENT_STATS elapsed 6.589µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:27.783+00:00 [Info] [Queryport ":9101"] connection "172.31.29.9:49110" doReceive() ... 2021-08-16T13:47:27.857+00:00 [Info] LifecycleMgr.handlePrepareCreateIndex() : Accept 12750664089235961203 2021-08-16T13:47:27.857+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_PREPARE_CREATE_INDEX elapsed 157.151µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:27.872+00:00 [Info] clustMgrAgent::OnIndexCreate Notification Received for Create Index DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false &{0} partitions [0] 2021-08-16T13:47:27.872+00:00 [Info] Indexer::handleCreateIndex InstId: 633044510811627291 Defn: DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false State: INDEX_STATE_CREATED RState: RebalActive Stream: NIL_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} 2021-08-16T13:47:27.872+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:27.872+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:27.872+00:00 [Info] Directory /opt/couchbase/var/lib/couchbase/data/@2i/shards does not exist. Skip shard recovery. 2021-08-16T13:47:27.872+00:00 [Info] Shard /opt/couchbase/var/lib/couchbase/data/@2i/shards/shard1(1) : Shard Created Successfully unexpected fault address 0x4004c67b1f fatal error: fault 2021-08-16T13:47:27.872+00:00 [Info] Shard /opt/couchbase/var/lib/couchbase/data/@2i/shards/shard1(1) : metadata saved successfully [signal SIGBUS: bus error code=0x1 addr=0x4004c67b1f pc=0xbd8ea4] goroutine 479 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40001069f0 sp=0x40001069c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4000106a30 sp=0x40001069f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400543c820, 0x4005820000, 0x11a9890, 0x4000106b40, 0x400543c838, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4000106ab0 sp=0x4000106a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400543c820, 0x4005820000, 0x11a9890, 0x4000106b40, 0x400543c838, 0x84, 0xc03eb8fbf4048cb0, 0x601d941fe) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4000106af0 sp=0x4000106ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000033a40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4000106da0 sp=0x4000106af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000033a40, 0x400528f730, 0x68, 0x100, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4000106e60 sp=0x4000106da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40056ed2b0, 0x40056ed2c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4000107200 sp=0x4000106e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40056ed2b0, 0x40056ed2c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40001078f0 sp=0x4000107200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x40058089f0, 0x40056e9500, 0x40051bb080, 0x40056ed290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4000107fb0 sp=0x40001078f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4000107fb0 sp=0x4000107fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [semacquire]: sync.runtime_Semacquire(0x40058089f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x40058089f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40051bb080, 0x4004396d50, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff8340969, 0x29, 0xfffff83409cb, 0x25, 0x4005728840, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4005621f70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff8340969, 0x29, 0xfffff83409cb, 0x25, 0x4005728840, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4005621f70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x40001063d0, 0x40001060a0, 0x4004396d50, 0x40002f2900, 0x40001785c0, 0x1, 0x7f, 0x4000105fe8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400133a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4005621f70, 0xb, 0x112ff8f, 0x6, 0x4005621f80, 0xd, 0x40050e7100, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).handleCreateIndex(0x400133a000, 0x12da5c8, 0x4005770580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1690 +0x60c github.com/couchbase/indexing/secondary/indexer.(*indexer).handleAdminMsgs(0x400133a000, 0x12da5c8, 0x4005770580, 0x0, 0x4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1553 +0x154 github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:992 +0x478 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb440, 0x4, 0x1136a87, 0xf, 0x4000107e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff89a53098, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8e00, 0x400235b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8e00, 0x400235b000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e008, 0x400235b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4001334060, 0x4000348001, 0xdff, 0xdff, 0x40002958e8, 0x6902d8, 0x4000295948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e048, 0x4000348001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000342000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000342000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000342000, 0xf5a960, 0x40013860e0, 0x6f0d60, 0x4004955e68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40013860c0, 0x4000046220, 0x4000295b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c050, 0x12f8f98, 0x40013860c0, 0x4004a5d617, 0x16, 0x40013cf7a0, 0x40000a3c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c050, 0x12f8f98, 0x40013860c0, 0x400008c050, 0x400033a070, 0x400033a080, 0x40000f8000, 0x4000046020, 0xee0980, 0x40013cf7a0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c050, 0x12f8f98, 0x40013860c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x40000a3f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000295f78, 0x40000fa3c0, 0x12db8a8, 0x400000ed20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b9180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000ad00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 20 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001354b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 68 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x12c18bc, 0x1, 0x40000a3e58, 0x40000385a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400131aec8, 0x40000385a0, 0x4001314780, 0x4000038658) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x4714f4, 0x4000038600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a3fa0, 0x4001344000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000178900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 66 [select]: net/http.(*persistConn).readLoop(0x400134e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: net/http.(*persistConn).writeLoop(0x400134e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000178900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52fb0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001356100, 0x400235c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001356100, 0x400235c000, 0x1000, 0x1000, 0x40013b8a88, 0x68f7a4, 0x40013ae050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e010, 0x400235c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400134e120, 0x400235c000, 0x1000, 0x1000, 0xf12180, 0x40013b0040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013340c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013340c0, 0xa, 0x1008880, 0x40013b0000, 0x199, 0x1000, 0x40013b8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013340c0, 0x2, 0x0, 0x9, 0xd5d, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400009ed50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400009ed50, 0x4000336000, 0xe00, 0xe00, 0xb4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013a4000, 0x4000336000, 0xe00, 0xe00, 0xb4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013a4000, 0x4000336000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013a4040, 0x4000336000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40013ae000, 0x0, 0x40013b8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40013ae000, 0x0, 0x0, 0x40013b8f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40013ae000, 0xee0d40, 0x40013b0000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4001394000, 0x4000082780, 0x40013ac000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 65 [select]: net/http.(*persistConn).readLoop(0x400428a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52ec8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237e080, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237e080, 0x400001c000, 0x1000, 0x1000, 0x400236dbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e060, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002a4a20, 0x400001c000, 0x1000, 0x1000, 0x400236dcf8, 0x67867c, 0x4001336ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013343c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40013343c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40002a4a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 73 [select]: net/http.(*persistConn).writeLoop(0x40002a4a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 57 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001785c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52870, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237e318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237e300, 0x40002a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237e300, 0x40002a1000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc4b8, 0x40002a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002a4fc0, 0x40002a1000, 0x1000, 0x1000, 0x400010d080, 0x200000003, 0x40000a77f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013ecf00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013ecf00, 0x40a, 0xffff604a0100, 0x100100000000, 0xffff6063afff, 0x400, 0x4004ca0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013ecf00, 0x41cff4, 0xffff604a0000, 0x100100000000, 0xffff6063afff, 0xffff60743b50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000303230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000303230, 0x400428d000, 0x1000, 0x1000, 0x40000a79b8, 0x0, 0x40000a7988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013a5580, 0x400428d000, 0x1000, 0x1000, 0x4002347800, 0x3, 0x40000a7be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013a5580, 0x400428d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013a55c0, 0x400428d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a7dd0, 0xa, 0x400428d201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a7dd0, 0x400428d20a, 0x1, 0x4004a5d7bb, 0x0, 0x400428d263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a7dd0, 0x4004c9210a, 0x4004a5d7bb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c3a40, 0x400139fbc0, 0x24, 0x40000a7f20, 0x4001399b90, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40013c3a40, 0x113075a, 0x7, 0x4001399b90, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40013c3a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 61 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52788, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ead18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013ead00, 0x4000329000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013ead00, 0x4000329000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc568, 0x4000329000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428a000, 0x4000329000, 0x1000, 0x1000, 0x400010d200, 0x200000003, 0x400010d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013ed740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013ed740, 0x40013c400a, 0x400029b8c8, 0x4d3b10, 0x40000820c0, 0x40013c4000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013ed740, 0x0, 0x1, 0x41a0a8, 0x40000820d8, 0x40013c4000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023b4450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023b4450, 0x40023c1000, 0x1000, 0x1000, 0x400029b9b8, 0x0, 0x400029b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000178a80, 0x40023c1000, 0x1000, 0x1000, 0x4002346600, 0x3, 0x400029bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000178a80, 0x40023c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000178ac0, 0x40023c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400029bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400029bdd0, 0xa, 0x40023c1101, 0x1, 0xe85, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400029bdd0, 0x40023c110a, 0x1, 0x40051f9969, 0x0, 0x40023c117b, 0x1, 0xe85, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400029bdd0, 0x40052ae20a, 0x40051f9969, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c3a40, 0x40013a3248, 0x17, 0x400029bf20, 0x4001399ba8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40013c3a40, 0x113075a, 0x7, 0x4001399ba8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40013c3a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff6075ca98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8680, 0x40042da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8680, 0x40042da000, 0x1000, 0x1000, 0x40042cfbe0, 0x469b10, 0x4004917800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105a0, 0x40042da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40013ce240, 0x40042da000, 0x1000, 0x1000, 0x40042cfcf8, 0x67867c, 0x40042d81e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40042a4720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40042a4720, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40013ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).readLoop(0x40002a4fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x40002a4fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 77 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 79 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 88 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000179000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff89a526a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eaf18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eaf00, 0x4004290000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eaf00, 0x4004290000, 0x1000, 0x1000, 0xe, 0x400033ec00, 0x23) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc588, 0x4004290000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428a120, 0x4004290000, 0x1000, 0x1000, 0x40003563b0, 0x40003562c0, 0x4001386158) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013edaa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013edaa0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013edaa0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013388d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013388d0, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001355640, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001355640, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001355680, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40023fd600, 0x40013dfe00, 0x4004364eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40023fd600, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40023fd600, 0xee0d40, 0x400135eaf0, 0x12db068, 0x4001355680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4001394f30, 0x4001386a80, 0x40013ac420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 119 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40013e6640, 0x40013dfd40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x400428a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [select]: net/http.(*persistConn).readLoop(0x400428a120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: net/http.(*persistConn).writeLoop(0x400428a120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 37 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000178840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40002b7da0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 39 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40002f89f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4001356200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff89a525b8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001356300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001356300, 0x4004dbbe60, 0x400236ae88, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40013523f0, 0x400236aef0, 0x400236aef8, 0x18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40013523f0, 0x11a95b0, 0x4001356280, 0x1303898, 0x400135af68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4001356280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000a04e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000a04e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000a04e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40002f94d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 183 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 145 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 48 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000176770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 178 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c7e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eb098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eb080, 0x4000337000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eb080, 0x4000337000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc5a0, 0x4000337000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428b0e0, 0x4000337000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023952c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023952c0, 0x100a, 0x0, 0x0, 0x400235ebe8, 0x597c30, 0x400235ec38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023952c0, 0x412b8c, 0x400235ec78, 0x655784, 0x400235eca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004902480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004902480, 0x40023f5a00, 0x200, 0x200, 0x4, 0x0, 0x400235ed01) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001355bc0, 0x40023f5a00, 0x200, 0x200, 0x40013acae7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001355bc0, 0x40023f5a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001355c00, 0x40023f5a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004900840, 0x680d00, 0x400235eeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004900840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004900840, 0xee0d40, 0x400135ed20, 0x12db068, 0x4001355c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40013955f0, 0x4001387440, 0x40013acea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 21 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013a4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52958, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ea318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013ea300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013ea300, 0x98f764, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400033cb70, 0x400133ec10, 0x1303898, 0x400133ec10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400033cb70, 0x1303898, 0x400133ec10, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40013a4d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 23 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40013a4ec0, 0x40013a8930, 0x40013a89a0, 0x4004288f00, 0x12e8f90, 0x40000f0000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40013960d4, 0x5, 0x40013a4d00, 0x12eca70, 0x40013a4b80, 0x130d478, 0x40000e80e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000f01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000f01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 24 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*metaNotifier).OnIndexCreate(0x400033cb88, 0x4004e492c0, 0x8c906a0e9f01f1b, 0x0, 0x4005672040, 0x1, 0x1, 0x4005672048, 0x1, 0x1, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:708 +0x260 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).CreateIndex(0x40000f0000, 0x4004e492c0, 0x4005672000, 0x4005672000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:1463 +0xaac github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).CreateIndexOrInstance(0x40000f0000, 0x4004e492c0, 0x400565e200, 0x4005672000, 0x4005647c00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:1370 +0x1d0 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processCommitToken(0x40000f0000, 0xb0f3756ee93d3573, 0x4005644de0, 0x0, 0x0, 0x12da228, 0x4005644e10, 0x1f0, 0x40056525a8, 0x400564d600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:932 +0x1b8 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).handleCommitCreateIndex(0x40000f0000, 0x40056546c0, 0x4004d57c78, 0x40056546c0, 0x0, 0x0, 0x1ba2100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:737 +0x1ac github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).handleCommit(0x40000f0000, 0x400564d600, 0x1f0, 0x200, 0x4004d57e28, 0x3, 0x3, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:690 +0x32c github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).dispatchRequest(0x40000f0000, 0x400566a280, 0x4004d57f10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:497 +0xd34 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:371 +0x324 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 25 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2659 +0x160 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 26 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002399710, 0x400035b6e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000178fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40013e6640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 29 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40013e6640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40013cc1d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 31 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40013e66e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 32 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40013e66e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 33 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40013a4ec0, 0x4004288f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 120 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x4000299e38, 0x40013dfe60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400006ce90, 0x40013dfe60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000299f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40023db080, 0x400006ce90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400237aa00, 0xd00007, 0x40023d77e8, 0x5, 0x400006ceb0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff89a523e8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001356800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001356800, 0x0, 0x40013b2d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001398558, 0x611a6c62, 0x40013b2db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001398558, 0x40013b2e20, 0x40013b2e28, 0x18, 0x4004394480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40023800e0, 0x12ed9d0, 0x4001398558, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40023800e0, 0x12ed9d0, 0x4001398558, 0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 124 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52300, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001356900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001356900, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001398570, 0xffff89ba1a68, 0xffff6071b578, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001398570, 0x665e1c, 0x0, 0xfe5e60, 0x40013b3dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4001398588, 0x4004396030, 0x11ae500, 0x40023805c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4002380540, 0x12ec710, 0x4001398588, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4002380540, 0x12ec710, 0x4001398588, 0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 182 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 104 [IO wait]: internal/poll.runtime_pollWait(0xffff6075cb80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8580, 0x40042d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8580, 0x40042d0000, 0x1000, 0x1000, 0x1900e48a127, 0x0, 0x40000a5a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010550, 0x40042d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x400429c7b0, 0x40042d0000, 0x1000, 0x1000, 0xab8f79a20, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x40042a43c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40042a43c0, 0x4, 0xab8f79a20, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x40000b9220, 0x12f86a0, 0x4000081b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 181 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 90 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52218, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237eb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237eb80, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237eb80, 0x40023f9000, 0x1000, 0x1000, 0x191799c82e0, 0x0, 0x40048a1a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135a280, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4001338b70, 0x40023f9000, 0x1000, 0x1000, 0xc2448a9ed, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x4001386c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4001386c00, 0x4, 0xc2448a9ed, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x4001361220, 0x12f86a0, 0x40013556c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 179 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40048b2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x40013ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c8c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ea798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013ea780, 0x40023ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013ea780, 0x40023ff000, 0x1000, 0x1000, 0x40023bebe0, 0x469b10, 0x4002347380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135a368, 0x40023ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428ad80, 0x40023ff000, 0x1000, 0x1000, 0x40023becf8, 0x67867c, 0x40013ac660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001386d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4001386d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400428ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 228 [select]: net/http.(*persistConn).writeLoop(0x400428b200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x400428ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40013a5fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 166 [select]: net/http.(*persistConn).writeLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40048b0160, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40048b0160, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40048b0160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 180 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 157 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 158 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x40048c3e38, 0x4002382c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004289a50, 0x4002382c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40048c3f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40023943c0, 0x4004289a50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 161 [select]: net/http.(*persistConn).readLoop(0x400428b0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [select]: net/http.(*persistConn).writeLoop(0x400428b0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 184 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 186 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9716 +0x474 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 187 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 188 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1025 +0xfc created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 339 [select]: net/http.(*persistConn).writeLoop(0x4004d3a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 191 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 192 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c610, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237f218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237f200, 0x40043ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237f200, 0x40043ab000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e300, 0x40043ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40023db680, 0x40048e2601, 0x5ff, 0x5ff, 0x400239f888, 0x6902d8, 0x400239f8e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4001398a20, 0x40048e2601, 0x5ff, 0x5ff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400439f1e0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400439f1e0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400439f1e0, 0xf5a960, 0x40023dbc40, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40023dbc20, 0x4002375fa0, 0x400239fac8, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400237ac30, 0x12f8f98, 0x40023dbc20, 0x4004b720a0, 0x16, 0x199, 0x400492fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400237ac30, 0x12f8f98, 0x40023dbc20, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356b00, 0x40013ab540, 0xfb91e0, 0x4002375f80, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400237ac30, 0x12f8f98, 0x40023dbc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004903470, 0x400492ff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400239ff10, 0x4004903470, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4000359300, 0x0, 0x0, 0x40023b9f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 328 [IO wait]: internal/poll.runtime_pollWait(0xffff6075bfb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ce0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ce0100, 0x4004cf4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ce0100, 0x4004cf4000, 0x1000, 0x1000, 0x19179a15b4f, 0x0, 0x4004cffa98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135a980, 0x4004cf4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4004cdae70, 0x4004cf4000, 0x1000, 0x1000, 0xc244798c1, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x4004cd2660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cd2660, 0x4, 0xc244798c1, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x4004a595e0, 0x12f86a0, 0x4004cde580) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 196 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*janitor).run(0x4001385b30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4377 +0xac created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:244 +0x2e8 goroutine 197 [select]: github.com/couchbase/indexing/secondary/manager.(*builder).run.func1(0x4001385980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4452 +0xa0 github.com/couchbase/indexing/secondary/manager.(*builder).run(0x4001385980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4459 +0xa4 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:247 +0x30c goroutine 198 [select]: github.com/couchbase/indexing/secondary/manager.(*updator).run(0x40013b0280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4902 +0x148 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:250 +0x330 goroutine 199 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processParallelsRequests(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:408 +0x10c created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:253 +0x350 goroutine 203 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x4004931e38, 0x400035b560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000357560, 0x400035b560, 0x11ae420, 0x400004a800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0xd96838, 0x40042c6f38) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004931f80, 0xd96838, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40013ecc60, 0x4000357560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 201 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x113ba6f, 0x14, 0x400493ff38, 0x4002383d40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x113ba6f, 0x14, 0x400493ff98, 0x4002383d40, 0x40048c8fb8, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).listenMoveIndex(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2297 +0xd8 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).recoverRebalance /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1966 +0x88 goroutine 202 [sleep]: time.Sleep(0x6fc23ac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).rebalanceJanitor(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1287 +0x40 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).recoverRebalance /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1967 +0xa8 goroutine 230 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c6f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eb418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eb400, 0x40048eb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eb400, 0x40048eb000, 0x1000, 0x1000, 0x1909ec02ceb, 0x0, 0x4004297a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc7e8, 0x40048eb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4004a4e1e0, 0x40048eb000, 0x1000, 0x1000, 0xb4971dbc3, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x40048f0720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40048f0720, 0x4, 0xb4971dbc3, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x40013e7900, 0x12f86a0, 0x40049ef2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 167 [select]: net/http.(*persistConn).readLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 206 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x4004941e38, 0x400035b4a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000357610, 0x400035b4a0, 0x4001319728, 0xd8d094) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x4000000004) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004941f80, 0x4714f4, 0x1be18c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40013ecc00, 0x4000357610) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: net/http.(*persistConn).readLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 293 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004abad00, 0x8, 0x9, 0x4004ad0480, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetTaskList(0x4000359300, 0x4004abad00, 0x8, 0x9, 0x4004ad0480, 0xadb658, 0x40048dda28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:338 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1(0x4004ad0480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:99 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40048ddb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetTaskList(0x13007d8, 0x4000359300, 0x4004abad00, 0x8, 0x9, 0x7530, 0x4004ac6a80, 0x46b478, 0x4004abc708) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:98 +0x94 reflect.Value.call(0x40023dbaa0, 0x400133e398, 0x13, 0x112ec54, 0x4, 0x40048ddf00, 0x3, 0x3, 0x400235feb8, 0x6a1f00, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023dbaa0, 0x400133e398, 0x13, 0x400235ff00, 0x3, 0x3, 0x4004aa0b00, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356c00, 0x40013ab740, 0xfb9280, 0x4004ab4500, 0x199, 0xee10c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 348 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c0a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ce0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ce0280, 0x4004e05000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ce0280, 0x4004e05000, 0x1000, 0x1000, 0x4004935688, 0x69f178, 0x40049356f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cccf0, 0x4004e05000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004caf7a0, 0x4004e05000, 0x1000, 0x1000, 0x5, 0x4005686f00, 0x40049357a8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d72fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d72fc0, 0x40a, 0xffff604f5e00, 0x100100000000, 0xffff6063afff, 0x400, 0x4004935810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d72fc0, 0x41cff4, 0xffff604f5d00, 0x100100000000, 0xffff6063afff, 0xffff603d3f30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d9a5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d9a5a0, 0x4004da0000, 0x1000, 0x1000, 0x4004935968, 0x0, 0x4004935938) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004cdec80, 0x4004da0000, 0x1000, 0x1000, 0x4002346600, 0x3, 0x4004935b90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004cdec80, 0x4004da0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004cdecc0, 0x4004da0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004935d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004935d80, 0xa, 0x4004da0201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004935d80, 0x4004da020a, 0x1, 0x40056f534e, 0x0, 0x4004da0259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004935d80, 0x400571ec0a, 0x40056f534e, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c3a40, 0x4004961ca0, 0x1f, 0x4004935ee0, 0x4004c9bec0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40013c3a40, 0x113075a, 0x7, 0x4004d74270, 0xd, 0x4004c9bec0, 0x0, 0xfb2801, 0x1c1bd01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40013c3a00, 0x4004d74270, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 227 [select]: net/http.(*persistConn).readLoop(0x400428b200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 211 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorMemUsage(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8873 +0x11c created by github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6788 +0x534 goroutine 212 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*indexer).logMemstats(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9159 +0x12c created by github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6789 +0x554 goroutine 216 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c528, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237f318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237f300, 0x40048d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237f300, 0x40048d5000, 0x1000, 0x1000, 0x40042cca38, 0x46b37c, 0x40042ccac8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc5b8, 0x40048d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d9e0, 0x40048d5000, 0x1000, 0x1000, 0x0, 0x400490e2a0, 0x40042ccbe8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002395620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002395620, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002395620, 0x41ab8c, 0xffff60485b00, 0x100100000000, 0xffff6063afff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004903c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004903c20, 0x4004910600, 0x200, 0x200, 0x40048ba508, 0x41cff4, 0xffff60485b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40049460c0, 0x4004910600, 0x200, 0x200, 0xffff6071ccd8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40049460c0, 0x4004910600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004946100, 0x4004910600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004956000, 0x0, 0x40048ba6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004956000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004956000, 0xee0d40, 0x400135f540, 0x12db068, 0x4004946100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004936900, 0x4004942f60, 0x4004938a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 214 [sleep]: time.Sleep(0xd18c2e2800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsPersister(0x40013e6640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3241 +0xe0 created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3035 +0x248 goroutine 217 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c270, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eb198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eb180, 0x40043b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eb180, 0x40043b8000, 0x1000, 0x1000, 0x40042cda88, 0x68f7a4, 0x40049561b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e318, 0x40043b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428b200, 0x40043b8000, 0x1000, 0x1000, 0xf12180, 0x400135f5d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023db8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023db8c0, 0xa, 0x1008880, 0x400135f590, 0x199, 0x1000, 0x40042cdb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023db8c0, 0x2, 0x0, 0x9, 0x1ba, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004903c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004903c80, 0x4004910800, 0x200, 0x200, 0x57, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004946140, 0x4004910800, 0x200, 0x200, 0x57, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004946140, 0x4004910800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004946180, 0x4004910800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004956160, 0x0, 0x40042cdeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004956160, 0x0, 0x0, 0x40042cdf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004956160, 0xee0d40, 0x400135f590, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004936990, 0x4004942fc0, 0x4004938a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 285 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004b4d0d0, 0x4004b5ec60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 173 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c440, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237f418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237f400, 0x40048d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237f400, 0x40048d3000, 0x1000, 0x1000, 0x40048e4360, 0xadb658, 0x40048c8a28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc5b0, 0x40048d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000db00, 0x40048d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002395500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002395500, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40042c43c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002395500, 0xee10c0, 0x40042c4401, 0x655784, 0x40042c4608, 0x498444) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40048ab5f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40048ab5f0, 0x4000317800, 0x200, 0x200, 0x4, 0x0, 0x2) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40048b23c0, 0x4000317800, 0x200, 0x200, 0x40042c45e8, 0x412bc0, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40048b23c0, 0x4000317800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40048b2400, 0x4000317800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40048d8840, 0x45a300, 0x40042c46b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40048d8840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40048d8840, 0xee0d40, 0x40013b16d0, 0x12db068, 0x40048b2400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400238f320, 0x4002395da0, 0x40048e4420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 284 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004b4d0c0, 0x8, 0x9, 0x4004b5ec60, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetCurrentTopology(0x4000359300, 0x4004b4d0c0, 0x8, 0x9, 0x4004b5ec60, 0xadb658, 0x40043b6a28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:390 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1(0x4004b5ec60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:76 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40043b6b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology(0x13007d8, 0x4000359300, 0x4004b4d0c0, 0x8, 0x9, 0x7530, 0x400237bea0, 0x46b478, 0x40042e3688) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:75 +0x94 reflect.Value.call(0x40023db9e0, 0x400133e358, 0x13, 0x112ec54, 0x4, 0x40043b6f00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023db9e0, 0x400133e358, 0x13, 0x40042c4700, 0x3, 0x3, 0x40042a5500, 0x40048add08, 0x40042c4784) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356b00, 0x4002374e80, 0xfb91e0, 0x4002375f40, 0x199, 0xee1180, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 338 [IO wait]: internal/poll.runtime_pollWait(0xffff89a524d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ebc98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013ebc80, 0x4004d62000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013ebc80, 0x4004d62000, 0x1000, 0x1000, 0x4004b19be0, 0x469b10, 0x4002347680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013ccb98, 0x4004d62000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d3a360, 0x4004d62000, 0x1000, 0x1000, 0x4004b19cf8, 0x67867c, 0x4004d60120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004ad39e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004ad39e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004d3a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 288 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004b4d160, 0x8, 0x9, 0x4004b5ed80, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetCurrentTopology(0x4000359300, 0x4004b4d160, 0x8, 0x9, 0x4004b5ed80, 0xadb658, 0x40043b6a28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:390 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1(0x4004b5ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:76 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40043b6b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology(0x13007d8, 0x4000359300, 0x4004b4d160, 0x8, 0x9, 0x7530, 0x4004b720a0, 0x46b478, 0x40042e3748) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:75 +0x94 reflect.Value.call(0x40023db9e0, 0x400133e358, 0x13, 0x112ec54, 0x4, 0x40043b6f00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023db9e0, 0x400133e358, 0x13, 0x4004366700, 0x3, 0x3, 0x40043666e8, 0x4bcf10, 0x4004b72058) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356b00, 0x40013ab540, 0xfb91e0, 0x4002375f80, 0x199, 0xee1180, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 289 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004b4d170, 0x4004b5ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 376 [select]: net/http.(*persistConn).writeLoop(0x4004caf7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 467 [select]: net/http.(*persistConn).writeLoop(0x40052fcc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 298 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004abada0, 0x4004ad05a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 294 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004abad10, 0x4004ad0480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 297 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004abad90, 0x8, 0x9, 0x4004ad05a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetTaskList(0x4000359300, 0x4004abad90, 0x8, 0x9, 0x4004ad05a0, 0xadb658, 0x40048dda28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:338 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1(0x4004ad05a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:99 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40048ddb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetTaskList(0x13007d8, 0x4000359300, 0x4004abad90, 0x8, 0x9, 0x7530, 0x4004ac6b70, 0x46b478, 0x4004abc7c8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:98 +0x94 reflect.Value.call(0x40023dbaa0, 0x400133e398, 0x13, 0x112ec54, 0x4, 0x40048ddf00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023dbaa0, 0x400133e398, 0x13, 0x400235ff00, 0x3, 0x3, 0x400235fee8, 0x4bcf10, 0x4004aac6e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356c00, 0x40023752c0, 0xfb9280, 0x4004ab4540, 0x199, 0xee10c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 375 [select]: net/http.(*persistConn).readLoop(0x4004caf7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 414 [select]: github.com/couchbase/indexing/secondary/queryport.(*Server).doPing(0x4001356280, 0x40051226c0, 0x4005113440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:348 +0xf8 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:265 +0x1c0 goroutine 453 [select]: net/http.(*persistConn).writeLoop(0x4004c2d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 466 [IO wait]: internal/poll.runtime_pollWait(0xffff60688a98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2f898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2f880, 0x4005723000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2f880, 0x4005723000, 0x1000, 0x1000, 0x40048c9be0, 0x469b10, 0x4002346600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133ec00, 0x4005723000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40052fcc60, 0x4005723000, 0x1000, 0x1000, 0x40048c9cf8, 0x67867c, 0x400555dec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40056bf2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40056bf2c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40052fcc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 452 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c188, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2ed98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2ed80, 0x40052cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2ed80, 0x40052cd000, 0x1000, 0x1000, 0x4004b18be0, 0x469b10, 0x40048f3800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e9c8, 0x40052cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c2d7a0, 0x40052cd000, 0x1000, 0x1000, 0x4004b18cf8, 0x67867c, 0x4005113260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40052c2a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40052c2a80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c2d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 470 [chan receive]: github.com/couchbase/gometa/common.(*PeerPipe).doSend(0x400547d080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:165 +0x64 created by github.com/couchbase/gometa/common.NewPeerPipe /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:63 +0x114 goroutine 413 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c9b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400515a518, 0x72, 0x0, 0x10000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400515a500, 0x4005160000, 0x6, 0x10000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400515a500, 0x4005160000, 0x6, 0x10000, 0x4005055180, 0x400515a580, 0x4005160000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135af68, 0x4005160000, 0x6, 0x10000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 github.com/couchbase/indexing/secondary/transport.fullRead(0xffff60688d58, 0x400135af68, 0x4005160000, 0x6, 0x10000, 0x4005160000, 0x6) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:189 +0x90 github.com/couchbase/indexing/secondary/transport.Receive(0xffff60688d58, 0x400135af68, 0x4005160000, 0x10000, 0x10000, 0x0, 0x60, 0x0, 0x400492fdf8, 0x4108e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:60 +0x60 github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0x400492ff78, 0xffff60688d58, 0x400135af68, 0xffff60688d58, 0x400135af68, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:119 +0x4c github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0x4001356280, 0x1303898, 0x400135af68, 0x40051226c0, 0x4005113440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:300 +0x25c created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:264 +0x190 goroutine 412 [chan receive]: github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0x4001356280, 0x1303898, 0x400135af68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:272 +0x1f8 created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:224 +0x2d4 goroutine 471 [IO wait]: internal/poll.runtime_pollWait(0xffff6075bd00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2fd98, 0x72, 0x0, 0x8, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2fd80, 0x4005621d90, 0x8, 0x8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2fd80, 0x4005621d90, 0x8, 0x8, 0x0, 0x736aa8, 0x4004e0fe08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133ec10, 0x4005621d90, 0x8, 0x8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0x400547d080, 0x8, 0x0, 0x0, 0x0, 0x1, 0x0, 0x298, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:284 +0x80 github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0x400547d080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:213 +0xb0 created by github.com/couchbase/gometa/common.NewPeerPipe /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:64 +0x134 goroutine 478 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 475 [select]: github.com/couchbase/gometa/protocol.(*messageListener).start(0x4005565770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:434 +0x164 created by github.com/couchbase/gometa/protocol.(*Leader).AddWatcher /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:276 +0x4c8 goroutine 477 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*metaNotifier).fetchStats(0x400033cb88) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:898 +0xa0 created by github.com/couchbase/indexing/secondary/indexer.(*metaNotifier).OnFetchStats /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:887 +0x3c goroutine 480 [runnable]: github.com/couchbase/plasma.(*StatsLogger).run(0x40000c5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/stats.go:1281 created by github.com/couchbase/plasma.(*StatsLogger).startOrStop /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/stats.go:1182 +0xfc Initializing write barrier = 8000 2021-08-16T13:47:28.066+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:28.066+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:28.074+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:28.079+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:28.079+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:28.079+00:00 [Info] Setting log level to Info 2021-08-16T13:47:28.079+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:28.082+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:28 audit: created new audit service 2021-08-16T13:47:28.125+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:28.125+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:28.125+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:28.125+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:28.126+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:28.159+00:00 [Info] security context initialized 2021-08-16T13:47:28.159+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:28.159+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:28.159+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:28.159+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:28.159+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:28.194+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:28.195+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:28.198+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:28.263+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:28.265+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:28.269+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:28.269+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:28.269+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:28.269+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:28.271+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:28.273+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:28.409+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:28.410+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:47:28.410+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:28.411+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:28.412+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:28.412+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:28.412+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:28.412+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:28.412+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:28.412+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:28.416+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.416+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.428+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:28.428+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:28.428+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:28.428+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.429+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:28.429+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:28.429+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:28.429+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:28.429+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:28.429+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:28.429+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:28.432+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_CREATED Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 2.696693ms 2021-08-16T13:47:28.432+00:00 [Warn] Indexer::validateIndexInstMap State INDEX_STATE_CREATED Stream NIL_STREAM Deferred true Found. Updating State to Ready InstId: 633044510811627291 Defn: DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false State: INDEX_STATE_CREATED RState: RebalActive Stream: NIL_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} 2021-08-16T13:47:28.432+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [ InstId: 633044510811627291 Defn: DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false State: INDEX_STATE_READY RState: RebalActive Stream: NIL_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} ] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:28.432+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:28.433+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.433+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:28.447+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_UPDATE_INDEX_INST elapsed 15.108482ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:28.468+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:28.468+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:28.468+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:28.468+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:28.468+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004f3025f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004f3025f pc=0xbd8ea4] goroutine 171 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40044069f0 sp=0x40044069c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004406a30 sp=0x40044069f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40000aad00, 0x400470b800, 0x11a9890, 0x4004406b40, 0x40000aad18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004406ab0 sp=0x4004406a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40000aad00, 0x400470b800, 0x11a9890, 0x4004406b40, 0x40000aad18, 0xffff92823b88, 0x4004406c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004406af0 sp=0x4004406ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004704500, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004406da0 sp=0x4004406af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004704500, 0x4001421110, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004406e60 sp=0x4004406da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c98c70, 0x4004c98c80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004407200 sp=0x4004406e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c98c70, 0x4004c98c80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40044078f0 sp=0x4004407200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004f3c170, 0x4004f2a380, 0x4004cfc580, 0x4004c98c50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004407fb0 sp=0x40044078f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004407fb0 sp=0x4004407fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000191920, 0x4, 0x1136a87, 0xf, 0x400010be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001a4108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001a4120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff92821248, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c800, 0x40001ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c800, 0x40001ff000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001940a8, 0x40001ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400018e840, 0x40000fa001, 0xdff, 0xdff, 0x40000a38e8, 0x6902d8, 0x4000107948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001a4168, 0x40000fa001, 0xdff, 0xdff, 0x40001079f8, 0x6f1968, 0x4000107938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000140000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000140000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000140000, 0xf5a960, 0x4000100560, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000100540, 0x40000462c0, 0x4000107b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400011c1e0, 0x12f8f98, 0x4000100540, 0x4000036be7, 0x16, 0x400000cfc0, 0x40000a3c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400011c1e0, 0x12f8f98, 0x4000100540, 0x400011c1e0, 0x4000112080, 0x4000112090, 0x400013e000, 0x4000114040, 0xee0980, 0x400000cfc0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400011c1e0, 0x12f8f98, 0x4000100540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000190720, 0x40000a3f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a9f78, 0x4000190720, 0x12db8a8, 0x40001a4138, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000190720, 0x40001b2500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000190720, 0x40001b2500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400011e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000557a0, 0x4, 0x4000055668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400008a060, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 9 [select]: net/http.(*persistConn).writeLoop(0x4000172120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019ae80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 8 [select]: net/http.(*persistConn).readLoop(0x4000172120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000192810, 0x12c18bc, 0x1, 0x40000a5e58, 0x4000038180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400017fec8, 0x4000038180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a5fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40000816c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40000816c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff92821160, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400013e198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400013e180, 0x40000fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400013e180, 0x40000fb000, 0x1000, 0x1000, 0x400012da88, 0x68f7a4, 0x400139a1b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010630, 0x40000fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172120, 0x40000fb000, 0x1000, 0x1000, 0xf12180, 0x400011c360, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400008a4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400008a4e0, 0xa, 0x1008880, 0x400011c320, 0x199, 0xad9, 0x400012db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400008a4e0, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003b6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003b6900, 0x40013aa000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400019a040, 0x40013aa000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400019a040, 0x40013aa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400019a080, 0x40013aa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400139a160, 0x0, 0x400012deb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400139a160, 0x0, 0x0, 0x400012df70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400139a160, 0xee0d40, 0x400011c320, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400016e120, 0x4000378d80, 0x40001c4240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff92821078, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8180, 0x4001398000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8180, 0x4001398000, 0x1000, 0x1000, 0x4000131be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x4001398000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4001398000, 0x1000, 0x1000, 0x4000131cf8, 0x67867c, 0x4000152cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000378c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000378c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 85 [select]: net/http.(*persistConn).writeLoop(0x40001f6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 84 [select]: net/http.(*persistConn).readLoop(0x40001f6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 86 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003ec5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 16 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 45 [select]: net/http.(*persistConn).readLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 46 [select]: net/http.(*persistConn).writeLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 48 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 49 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 29 [select]: net/http.(*persistConn).readLoop(0x40001f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [select]: net/http.(*persistConn).writeLoop(0x40001f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [IO wait]: internal/poll.runtime_pollWait(0xffff92820938, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400013e818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400013e800, 0x400139c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400013e800, 0x400139c000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044160a0, 0x400139c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001f6a20, 0x400139c000, 0x1000, 0x1000, 0x4004418480, 0x200000003, 0x4004418480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003feea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003feea0, 0x40013b000a, 0x40000a38c8, 0x4d3b10, 0x400008a0c0, 0x40013b0000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003feea0, 0x0, 0x1, 0x0, 0x400008a0d8, 0x40013b0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400140d350) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400140d350, 0x400443a000, 0x1000, 0x1000, 0x40000a39b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000118640, 0x400443a000, 0x1000, 0x1000, 0x400008a9b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000118640, 0x400443a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000118680, 0x400443a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a3dd0, 0xa, 0x400443a101, 0x1, 0xe85, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a3dd0, 0x400443a10a, 0x1, 0x40024051e9, 0x0, 0x400443a17b, 0x1, 0xe85, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a3dd0, 0x40001a710a, 0x40024051e9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001a6fb0, 0x4000138588, 0x17, 0x40000a3f20, 0x4000371188, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40001a6fb0, 0x113075a, 0x7, 0x4000371188, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40001a6f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 67 [IO wait]: internal/poll.runtime_pollWait(0xffff92820b08, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004430080, 0x4004434000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004430080, 0x4004434000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4004416008, 0x4004434000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001f6000, 0x4004434000, 0x1000, 0x1000, 0x4004418600, 0x200000003, 0x4004418600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fe480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fe480, 0x40013b000a, 0x400010b8c8, 0x4d3b10, 0x400008a0c0, 0x40013b0000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fe480, 0x0, 0x1, 0x0, 0x400008a0d8, 0x40013b0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400442acc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400442acc0, 0x400139e000, 0x1000, 0x1000, 0x400010b9b8, 0x0, 0x400010b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004410700, 0x400139e000, 0x1000, 0x1000, 0x4000120900, 0x3, 0x400010bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004410700, 0x400139e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004410740, 0x400139e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400010bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010bdd0, 0xa, 0x400139e201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400010bdd0, 0x400139e20a, 0x1, 0x4000037d5b, 0x0, 0x400139e263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400010bdd0, 0x400011440a, 0x4000037d5b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001a6fb0, 0x400011a420, 0x24, 0x400010bf20, 0x40003e4d20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40001a6fb0, 0x113075a, 0x7, 0x40003e4d20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40001a6f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffff928204b0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f9518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f9500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f9500, 0x4714f4, 0x40003714a0, 0x12ed9d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bf7bf0, 0x1, 0x12ed9d0, 0x40003714a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bf7bf0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004d18080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 75 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004411a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 68 [IO wait]: internal/poll.runtime_pollWait(0xffff92820a20, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8280, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8280, 0x400001e000, 0x1000, 0x1000, 0x40000a7688, 0x69f178, 0x40000a76f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110120, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d8c0, 0x400001e000, 0x1000, 0x1000, 0x5, 0x400243c000, 0x40000a77c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000101260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000101260, 0x68f90a, 0x4000174000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000101260, 0x41c300, 0xffff92824518, 0x0, 0x40000a7888, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400140d650) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400140d650, 0x4004445000, 0x1000, 0x1000, 0x40000a7998, 0x44e884, 0xffff92913f18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000118740, 0x4004445000, 0x1000, 0x1000, 0x400008a9b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000118740, 0x4004445000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000118780, 0x4004445000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a7d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a7d80, 0xa, 0x4004445201, 0x1, 0xffff68088cf8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a7d80, 0x400444520a, 0x1, 0x400452d390, 0x0, 0x4004445259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a7d80, 0x400244a00a, 0x400452d390, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001a6fb0, 0x40003e3ac0, 0x1f, 0x40000a7ee0, 0x40003e4ea0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40001a6fb0, 0x113075a, 0x7, 0x4002405200, 0xd, 0x40003e4ea0, 0x0, 0x4714f4, 0x40001a6fb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40001a6f70, 0x4002405200, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 87 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003ff260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400442afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400013e880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 90 [IO wait]: internal/poll.runtime_pollWait(0xffff92820768, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400013e998, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400013e980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400013e980, 0x0, 0x4000124e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40003714a0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40003714a0, 0x12ed9d0, 0x40003714a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400013e900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d520, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400013d520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 94 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400442baa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff92820680, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8580, 0x40003f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8580, 0x40003f2000, 0x1000, 0x1000, 0x400012abe0, 0x469b10, 0x4001395380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010558, 0x40003f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003ca6c0, 0x40003f2000, 0x1000, 0x1000, 0x400012acf8, 0x67867c, 0x4000038f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400008af60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400008af60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003ca6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004d184c0, 0x40001cb340, 0x40001cb3b0, 0x40045cb0d0, 0x12e8f90, 0x40000f0fc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045f7c44, 0x5, 0x4004d18080, 0x12eca70, 0x4000081f00, 0x130d478, 0x4000170310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400039eb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400039eb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x40003ca6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 105 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 97 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e8150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 113 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 107 [IO wait]: internal/poll.runtime_pollWait(0xffff92820598, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004430880, 0x4004452000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004430880, 0x4004452000, 0x1000, 0x1000, 0x4000130a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110168, 0x4004452000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x4004452000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x4000094280) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003793e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003793e0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40013bf438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003793e0, 0x0, 0x40013bf478, 0x655784, 0x40013bf4a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400447cb10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400447cb10, 0x400445f800, 0x200, 0x200, 0x60, 0x0, 0x40013bf501) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000119280, 0x400445f800, 0x200, 0x200, 0x40001c4a20, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000119280, 0x400445f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001192c0, 0x400445f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400477d340, 0x680d00, 0x40013bf6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400477d340, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400477d340, 0xee0d40, 0x4000094960, 0x12db068, 0x40001192c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400475ad80, 0x40047783c0, 0x4004c960c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044111c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 78 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000f0fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 79 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000f0fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 80 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000396fc0, 0x4004bfe5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 81 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400019af80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40013a5400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 131 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40013a5400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010b18) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 133 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40013a54a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 134 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40013a54a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 135 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004d184c0, 0x40045cb0d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 136 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40013a5400, 0x4004bfe6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 55 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000192810, 0x1144559, 0x1b, 0x4004621e38, 0x400244f260, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400010f0b0, 0x400244f260, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004621f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002430420, 0x400010f0b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 56 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 57 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400011c640, 0xd00007, 0x400460c068, 0x5, 0x400010f0c0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff92820850, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400019c200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400019c200, 0xffff92913108, 0x30, 0x4004bd7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001436bd0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001436bd0, 0x4004bd7dd0, 0x11ae500, 0x4002454160, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40024540e0, 0x12ed9d0, 0x4001436bd0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40024540e0, 0x12ed9d0, 0x4001436bd0, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 59 [IO wait]: internal/poll.runtime_pollWait(0xffff928203c8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400019c300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400019c300, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001436be8, 0xffff92913108, 0xffff5296ef58, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001436be8, 0x665e1c, 0x0, 0xfe5e60, 0x40013f2dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4001436c00, 0x4004bd7e60, 0x11ae500, 0x4002454240, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40024541c0, 0x12ec710, 0x4001436c00, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40024541c0, 0x12ec710, 0x4001436c00, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff529d4c18, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c780, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c780, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194348, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172d80, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002431800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002431800, 0xa, 0x1, 0x2, 0x1311860, 0x1, 0x40013bfc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002431800, 0x2, 0x40013bfc78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400447cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400447cea0, 0x400445fe00, 0x200, 0x200, 0x4, 0x0, 0x40013bfd08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000119400, 0x400445fe00, 0x200, 0x200, 0x400003e607, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000119400, 0x400445fe00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000119440, 0x400445fe00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400477db80, 0x1311800, 0x40013bfeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400477db80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400477db80, 0xee0d40, 0x4000094a00, 0x12db068, 0x4000119440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400475afc0, 0x4004778600, 0x4004c96600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 109 [IO wait]: internal/poll.runtime_pollWait(0xffff529d4de8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004430d80, 0x4004ca2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004430d80, 0x4004ca2000, 0x1000, 0x1000, 0x4001404be0, 0x469b10, 0x4004d6a180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110230, 0x4004ca2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40045fd440, 0x4004ca2000, 0x1000, 0x1000, 0x4001404cf8, 0x67867c, 0x4004c965a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004778540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004778540, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40045fd440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 112 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003ecb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 110 [select]: net/http.(*persistConn).writeLoop(0x40045fd440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 64 [semacquire]: sync.runtime_Semacquire(0x4004f3c178) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004f3c170) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004cfc580, 0x400140c3f0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff8b13969, 0x29, 0xfffff8b139cb, 0x25, 0x4004c97380, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004d1e9e0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff8b13969, 0x29, 0xfffff8b139cb, 0x25, 0x4004c97380, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004d1e9e0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004407190, 0x4004406e60, 0x400140c3f0, 0x40013b6fc0, 0x400019ae80, 0x0, 0x17f, 0x4004d74da8, 0xffff5011adc8, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000154000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004d1e9e0, 0xb, 0x4004d1e9d8, 0x6, 0x4004d1e9f0, 0xd, 0x4004bf8c60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000154000, 0x4000370000, 0x4000102360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000154000, 0x4004408d00, 0x2, 0x2, 0x4004408d10, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40003ecb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 116 [IO wait]: internal/poll.runtime_pollWait(0xffff529d4d00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c600, 0x4004b69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c600, 0x4004b69000, 0x1000, 0x1000, 0x4001400be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044162a0, 0x4004b69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172c60, 0x4004b69000, 0x1000, 0x1000, 0x4001400cf8, 0x67867c, 0x4004c241e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400455b380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400455b380, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000172c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000175080, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000175080, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4000175080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 155 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 156 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000192810, 0x1144559, 0x1b, 0x4004d77e38, 0x400244faa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400010f530, 0x400244faa0, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d77f80, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002430ae0, 0x400010f530) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 159 [select]: net/http.(*persistConn).readLoop(0x4000172d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 160 [select]: net/http.(*persistConn).writeLoop(0x4000172d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [select]: net/http.(*persistConn).writeLoop(0x4000172c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004cb6a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004cb6a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 170 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:33.671+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:33.671+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:33.681+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:33.683+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:33.683+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:33.683+00:00 [Info] Setting log level to Info 2021-08-16T13:47:33.683+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:33.686+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:33 audit: created new audit service 2021-08-16T13:47:33.695+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:33.695+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:33.695+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:33.695+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:33.695+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:33.706+00:00 [Info] security context initialized 2021-08-16T13:47:33.706+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:33.706+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:33.706+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:33.706+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:33.706+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:33.728+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:33.729+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:33.729+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:33.750+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:33.750+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:33.753+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:33.754+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:33.755+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:33.758+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:33.761+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:33.761+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:33.803+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:33.803+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:47:33.803+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:33.805+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:33.805+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:33.815+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:33.815+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:33.815+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:33.815+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:33.815+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:33.815+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.815+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.816+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:33.816+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:33.816+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:33.817+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.818+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:33.818+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:33.818+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:33.818+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:33.818+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:33.818+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:33.818+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:33.822+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.16691ms 2021-08-16T13:47:33.822+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:33.824+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.824+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004b4a6df 2021-08-16T13:47:33.835+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:33.835+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:33.835+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:33.835+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:33.835+00:00 [Info] : Recover shard shard1 from metadata completed. fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004b4a6df pc=0xbd8ea4] goroutine 181 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004edc9f0 sp=0x4004edc9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004edca30 sp=0x4004edc9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4000165a00, 0x4004b2ec00, 0x11a9890, 0x4004edcb40, 0x4000165a18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004edcab0 sp=0x4004edca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4000165a00, 0x4004b2ec00, 0x11a9890, 0x4004edcb40, 0x4000165a18, 0xffff7f6db758, 0x4004edcc18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004edcaf0 sp=0x4004edcab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40003e0a00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004edcda0 sp=0x4004edcaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003e0a00, 0x40001696c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004edce60 sp=0x4004edcda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c6c6c0, 0x4004c6c6d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004edd200 sp=0x4004edce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c6c6c0, 0x4004c6c6d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004edd8f0 sp=0x4004edd200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ec2df0, 0x4004c63500, 0x40044cbb80, 0x4004c6c6a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004eddfb0 sp=0x4004edd8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004eddfb0 sp=0x4004eddfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000204f60, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126080, 0x400013d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126080, 0x400013d000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114018, 0x400013d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e120, 0x400015c001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000124030, 0x400015c001, 0xdff, 0xdff, 0x40000f59f8, 0x6f1968, 0x40000f5938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400013a160, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400013a160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400013a160, 0xf5a960, 0x400010e500, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400010e4e0, 0x40001161c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400014c140, 0x12f8f98, 0x400010e4e0, 0x400011c8c7, 0x16, 0x4000130480, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400014c140, 0x12f8f98, 0x400010e4e0, 0x400014c140, 0x400011c0d0, 0x400011c0e0, 0x4000126100, 0x40001160e0, 0xee0980, 0x4000130480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400014c140, 0x12f8f98, 0x400010e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097ec0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a1f78, 0x4000097ec0, 0x12db8a8, 0x4000124000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097ec0, 0x40000b1360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097ec0, 0x40000b1360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 26 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x12c18bc, 0x1, 0x40001bfe58, 0x400012a720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000198ec8, 0x400012a720, 0x4000198ec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x16) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40001bffa0, 0x0, 0x400011a100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400011a240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 24 [select]: net/http.(*persistConn).readLoop(0x40003d67e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x40003d67e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400011a240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6552c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001f8498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001f8480, 0x40045b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001f8480, 0x40045b2000, 0x1000, 0x1000, 0x400237cbe0, 0x469b10, 0x4000197080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001880a0, 0x40045b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ce360, 0x40045b2000, 0x1000, 0x1000, 0x400237ccf8, 0x67867c, 0x400012b2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000186ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000186ea0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ce360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x400001e000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000103e8, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d6c60, 0x400001e000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x4000038a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a97a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003a97a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003d6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: net/http.(*persistConn).writeLoop(0x40003d6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x400015d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x400015d000, 0x1000, 0x1000, 0x40001b2a88, 0x68f7a4, 0x4000446050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100d0, 0x400015d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d67e0, 0x400015d000, 0x1000, 0x1000, 0xf12180, 0x400014c220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a9320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a9320, 0xa, 0x1008880, 0x400014c1e0, 0x199, 0x1000, 0x40001b2b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003a9320, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000434030) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000434030, 0x400044c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400043e000, 0x400044c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400043e000, 0x400044c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400043e040, 0x400044c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000446000, 0x0, 0x40001b2eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000446000, 0x0, 0x0, 0x40001b2f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000446000, 0xee0d40, 0x400014c1e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400042c000, 0x40000824e0, 0x4000444000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 74 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 89 [select]: net/http.(*persistConn).writeLoop(0x40001ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: net/http.(*persistConn).readLoop(0x40001ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 52 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400043ea40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 40 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000e82a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 82 [select]: net/http.(*persistConn).readLoop(0x40003d6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x40003d6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 54 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004623220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 131 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 60 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400043ef40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 61 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000417a40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40023d4d50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000127780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000127898, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000127880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000127880, 0x0, 0x4002375698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023fe540, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023fe540, 0x12ed9d0, 0x40023fe540, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000127800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400045bad0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400045bad0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400045bad0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40023d5830) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6553a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126280, 0x400462c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126280, 0x400462c000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114098, 0x400462c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000db00, 0x400462c000, 0x1000, 0x1000, 0x40043fa600, 0x200000003, 0x40043fa600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f3e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f3e0, 0x400039200a, 0x40001b98c8, 0x4d3b10, 0x40000820c0, 0x4000392000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f3e0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000392000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003af080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003af080, 0x4004630000, 0x1000, 0x1000, 0x40001b99b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044f82c0, 0x4004630000, 0x1000, 0x1000, 0x400010e598, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044f82c0, 0x4004630000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044f8300, 0x4004630000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001b9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001b9dd0, 0xa, 0x4004630201, 0x1, 0xd33, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001b9dd0, 0x400463020a, 0x1, 0x40044e5229, 0x0, 0x40046302cd, 0x1, 0xd33, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001b9dd0, 0x40000e5e0a, 0x40044e5229, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e4380, 0x40001509a8, 0x17, 0x40001b9f20, 0x40044f5a58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000e4380, 0x113075a, 0x7, 0x40044f5a58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000e4340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 73 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 103 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 94 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044f8b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001f8298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001f8280, 0x40003aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001f8280, 0x40003aa000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000188070, 0x40003aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ce240, 0x40003aa000, 0x1000, 0x1000, 0x40043fac00, 0x200000003, 0x40043fac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000186a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000186a80, 0x400039200a, 0x40001bd8c8, 0x4d3b10, 0x40000820c0, 0x4000392000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000186a80, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000392000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246c870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246c870, 0x40045b1000, 0x1000, 0x1000, 0x40001bd9b8, 0x0, 0x40001bd988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a580, 0x40045b1000, 0x1000, 0x1000, 0x4000196a80, 0x3, 0x40001bdbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a580, 0x40045b1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a5c0, 0x40045b1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001bddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bddd0, 0xa, 0x40045b1201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001bddd0, 0x40045b120a, 0x1, 0x4002472b6b, 0x0, 0x40045b1263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001bddd0, 0x40001c420a, 0x4002472b6b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e4380, 0x400011e120, 0x24, 0x40001bdf20, 0x40044f5a40, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000e4380, 0x113075a, 0x7, 0x40044f5a40, 0x0, 0x400014c140, 0x400011c0d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000e4340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6551d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126680, 0x4004640000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126680, 0x4004640000, 0x1000, 0x1000, 0x4004629688, 0x69f178, 0x40046296f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114118, 0x4004640000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000130120, 0x4004640000, 0x1000, 0x1000, 0x5, 0x40044b0a00, 0x40046297c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f9e0, 0x68f90a, 0x4000162b00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f9e0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246cd50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246cd50, 0x40045c8000, 0x1000, 0x1000, 0x4004629998, 0x44e884, 0x4004629948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a840, 0x40045c8000, 0x1000, 0x1000, 0x400010e598, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a840, 0x40045c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a880, 0x40045c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004629d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004629d80, 0xa, 0x40045c8201, 0x1, 0xffff7f653fd8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004629d80, 0x40045c820a, 0x1, 0x4000183c40, 0x0, 0x40045c8259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004629d80, 0x40000ce40a, 0x4000183c40, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e4380, 0x40044f6ec0, 0x1f, 0x4004629ee0, 0x400463e078, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000e4380, 0x113075a, 0x7, 0x40044e5240, 0xd, 0x400463e078, 0x0, 0x4714f4, 0x40000e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000e4340, 0x40044e5240, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 110 [select]: net/http.(*persistConn).readLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 111 [select]: net/http.(*persistConn).writeLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x40001ce360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400043e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [select]: net/http.(*persistConn).readLoop(0x4000130120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 116 [select]: net/http.(*persistConn).writeLoop(0x4000130120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004623220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001f8a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001f8a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001f8a00, 0x12d9d48, 0x400011b040, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb4630, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb4630, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400011ba00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 133 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40044f9d40, 0x40001c2d20, 0x40001c2d90, 0x400040da20, 0x12e8f90, 0x40000e82a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046bbcb4, 0x5, 0x400011ba00, 0x12eca70, 0x400011b880, 0x130d478, 0x4000208000, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 120 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40044f9d40, 0x400040da20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000e82a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 42 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000140120, 0x40023b4a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400043e100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40023a63c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 45 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40023a63c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000458410) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40023a63c0, 0x40023b4a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 48 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x1144559, 0x1b, 0x400462be38, 0x40023b4ba0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40023a2f70, 0x40023b4ba0, 0x4002375698, 0xccadf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12ed9d0, 0x40023fe540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400462bf80, 0x40023fe540, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400460b8c0, 0x40023a2f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 49 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400014cc30, 0xd00007, 0x4002442c18, 0x5, 0x40023a2f80, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6550f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1e00, 0xffff7f6d1108, 0x30, 0x4004ad9530) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023f9650, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023f9650, 0x4004ad9530, 0x11ae500, 0x4004466e80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4004466e00, 0x12ed9d0, 0x40023f9650, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4004466e00, 0x12ed9d0, 0x40023f9650, 0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1f18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1f00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023f9668, 0xffff7f6d1108, 0xffff7f66df38, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023f9668, 0x665e1c, 0x0, 0xfe5e60, 0x400461f5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40023f9680, 0x4004ad95c0, 0x11ae500, 0x4004466f60, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4004466ee0, 0x12ec710, 0x40023f9680, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4004466ee0, 0x12ec710, 0x40023f9680, 0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0400, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0400, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104a0, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d6d80, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a9ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a9ce0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400237ac38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003a9ce0, 0x0, 0x400237ac78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bfe720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bfe720, 0x4004bb1600, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081680, 0x4004bb1600, 0x200, 0x200, 0x4004be6647, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081680, 0x4004bb1600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000816c0, 0x4004bb1600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004dc46e0, 0x0, 0x400237aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004dc46e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004dc46e0, 0xee0d40, 0x40001caeb0, 0x12db068, 0x40000816c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bb9320, 0x40046fb5c0, 0x4004d80840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 150 [IO wait]: internal/poll.runtime_pollWait(0xffff540b1598, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d00098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d00080, 0x4004d02000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d00080, 0x4004d02000, 0x1000, 0x1000, 0x400460cbe0, 0x469b10, 0x4000103380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000458420, 0x4004d02000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001cf680, 0x4004d02000, 0x1000, 0x1000, 0x400460ccf8, 0x67867c, 0x40023b4d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400460b9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400460b9e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001cf680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 151 [select]: net/http.(*persistConn).writeLoop(0x40001cf680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 71 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 70 [semacquire]: sync.runtime_Semacquire(0x4004ec2df8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ec2df0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40044cbb80, 0x4000180300, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd6ea5969, 0x29, 0xffffd6ea59cb, 0x25, 0x4004ed4060, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd6a50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd6ea5969, 0x29, 0xffffd6ea59cb, 0x25, 0x4004ed4060, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd6a50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004edd190, 0x4004edce60, 0x4000180300, 0x40000b8d80, 0x400043ea40, 0x0, 0xffff54101350, 0x88, 0xffff7f6d1130, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400016e380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bd6a50, 0xb, 0x4004bd6a48, 0x6, 0x4004bd6a60, 0xd, 0x4004bafaa0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400016e380, 0x40003a0000, 0x4000110c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400016e380, 0x40004257a0, 0x2, 0x2, 0x40004257b0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 75 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40003d5340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 167 [IO wait]: internal/poll.runtime_pollWait(0xffff540b1680, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ca4698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ca4680, 0x4004e76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ca4680, 0x4004e76000, 0x1000, 0x1000, 0x4002445be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010770, 0x4004e76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e5c000, 0x4004e76000, 0x1000, 0x1000, 0x4002445cf8, 0x67867c, 0x40045a5da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004439e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004439e00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e5c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e384d0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e384d0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 80 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004e384d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 81 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 162 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x1144559, 0x1b, 0x4004e69e38, 0x40045a5080, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400441d340, 0x40045a5080, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e69f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044390e0, 0x400441d340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x4004e5c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: net/http.(*persistConn).readLoop(0x4004e5c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 166 [select]: net/http.(*persistConn).writeLoop(0x4004e5c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 121 [IO wait]: internal/poll.runtime_pollWait(0xffff540b14b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ca4798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ca4780, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ca4780, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010768, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e5c120, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004439ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004439ce0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400461f418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004439ce0, 0x412bc0, 0x0, 0x0, 0x400461f4c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ad9620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ad9620, 0x4004b45000, 0x200, 0x200, 0x4004ad9590, 0x30, 0x400461f578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044f9f00, 0x4004b45000, 0x200, 0x200, 0x0, 0xffff7f6d1108, 0x400461f568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044f9f00, 0x4004b45000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044f9f40, 0x4004b45000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b43b80, 0x12ec700, 0x400461f6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b43b80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b43b80, 0xee0d40, 0x4000190eb0, 0x12db068, 0x40044f9f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004b46870, 0x4004b2ade0, 0x4004adb7a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 122 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003d5380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 123 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 124 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 125 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 126 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c68600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c68600, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 180 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:39.004+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:39.005+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:39.012+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:39.014+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:39.014+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:39.014+00:00 [Info] Setting log level to Info 2021-08-16T13:47:39.014+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:39.017+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:39 audit: created new audit service 2021-08-16T13:47:39.024+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:39.024+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:39.024+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:39.024+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:39.024+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:39.036+00:00 [Info] security context initialized 2021-08-16T13:47:39.036+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:39.037+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:39.037+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:39.037+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:39.037+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:39.050+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:39.058+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:39.067+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:39.078+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:39.081+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:39.081+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:39.081+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:39.084+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:39.085+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:39.088+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:39.090+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:39.137+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:39.137+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 57 us 2021-08-16T13:47:39.138+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:39.145+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:39.146+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:39.147+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:39.147+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:39.147+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:39.147+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:39.147+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:39.149+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.149+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.150+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:39.150+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:39.150+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:39.150+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.151+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:39.151+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:39.151+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:39.151+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:39.151+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:39.151+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:39.151+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:39.154+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 2.951207ms 2021-08-16T13:47:39.154+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:39.155+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:39.155+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.155+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:39.156+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:39.156+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:39.156+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 2021-08-16T13:47:39.166+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:39.166+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:39.166+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:39.166+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:39.166+00:00 [Info] : Recover shard shard1 from metadata completed. 0x40001db45f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40001db45f pc=0xbd8ea4] goroutine 198 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d529f0 sp=0x4004d529c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d52a30 sp=0x4004d529f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40044bed00, 0x400046b800, 0x11a9890, 0x4004d52b40, 0x40044bed18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d52ab0 sp=0x4004d52a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40044bed00, 0x400046b800, 0x11a9890, 0x4004d52b40, 0x40044bed18, 0xffffb105cf60, 0x4004d52c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d52af0 sp=0x4004d52ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40003e2640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d52da0 sp=0x4004d52af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003e2640, 0x40044a5180, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d52e60 sp=0x4004d52da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dd0440, 0x4004dd0450, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d53200 sp=0x4004d52e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dd0440, 0x4004dd0450, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d538f0 sp=0x4004d53200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ddc8c0, 0x4004dcea80, 0x40000d1600, 0x4004dd0420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d53fb0 sp=0x4004d538f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d53fb0 sp=0x4004d53fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000204f90, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8872de88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a0080, 0x40001bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a0080, 0x40001bb000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194010, 0x40001bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400018e120, 0x40001ba001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x400044b948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400019c030, 0x40001ba001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001b8160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b8160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b8160, 0xf5a960, 0x40001144a0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000114480, 0x4000118160, 0x400044bb28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000126140, 0x12f8f98, 0x4000114480, 0x40001108b7, 0x16, 0x4000398240, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000126140, 0x12f8f98, 0x4000114480, 0x4000126140, 0x40001100c0, 0x40001100d0, 0x400011c000, 0x4000118000, 0xee0980, 0x4000398240, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000126140, 0x12f8f98, 0x4000114480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097ef0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40034cbf78, 0x4000097ef0, 0x12db8a8, 0x400019c000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097ef0, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097ef0, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 20 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f6080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 40 [select]: net/http.(*persistConn).writeLoop(0x4000399440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 37 [select]: net/http.(*persistConn).readLoop(0x40001c8120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x40001c8120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 52 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c300, 0x12c18bc, 0x1, 0x400009fe58, 0x4000038c60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400012d6c8, 0x4000038c60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40003dc740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 53 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40003dc740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff8872dda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0100, 0x40001ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0100, 0x40001ce000, 0x1000, 0x1000, 0x4000135a88, 0x68f7a4, 0x40001b8730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b8, 0x40001ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c8120, 0x40001ce000, 0x1000, 0x1000, 0xf12180, 0x40001262c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400018e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400018e4e0, 0xa, 0x1008880, 0x4000126280, 0x199, 0x1000, 0x4000135b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400018e4e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010ee40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010ee40, 0x400045c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a300, 0x400045c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a300, 0x400045c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a340, 0x400045c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001b86e0, 0x0, 0x4000135eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b86e0, 0x0, 0x0, 0x4000135f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b86e0, 0xee0d40, 0x4000126280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001a8cf0, 0x4000114a20, 0x4000038ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff8872dcb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c180, 0x4000458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c180, 0x4000458000, 0x1000, 0x1000, 0x4000132be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116180, 0x4000458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003990e0, 0x4000458000, 0x1000, 0x1000, 0x4000132cf8, 0x67867c, 0x4000038e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001148a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001148a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003990e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 57 [select]: net/http.(*persistConn).writeLoop(0x40003990e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [select]: net/http.(*persistConn).readLoop(0x40034fc240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).readLoop(0x4000399440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d000, 0x400450d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d000, 0x400450d000, 0x1000, 0x1000, 0x1, 0x0, 0x40001416f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116330, 0x400450d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40034fc240, 0x400450d000, 0x1000, 0x1000, 0x5, 0x40044d8000, 0x40001417c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004504480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004504480, 0x68f90a, 0x40001418a8, 0x412bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004504480, 0x0, 0x4000141868, 0x41a0a8, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400449ab40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400449ab40, 0x40044c7000, 0x1000, 0x1000, 0x4000141998, 0x44e884, 0x40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044a2440, 0x40044c7000, 0x1000, 0x1000, 0x4004492898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044a2440, 0x40044c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044a2480, 0x40044c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000141d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000141d80, 0xa, 0x40044c7201, 0x1, 0xffff882af418, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000141d80, 0x40044c720a, 0x1, 0x4004526c00, 0x0, 0x40044c7259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000141d80, 0x40044d260a, 0x4004526c00, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004484520, 0x4004465120, 0x1f, 0x4000141ee0, 0x4004466618, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4004484520, 0x113075a, 0x7, 0x4004526650, 0xd, 0x4004466618, 0x0, 0x400011a340, 0x40001b86e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40044844e0, 0x4004526650, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 82 [select]: net/http.(*persistConn).writeLoop(0x40034fc240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004484d00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 25 [select]: net/http.(*persistConn).readLoop(0x40023fc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x40023fc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 69 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a0d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a0d00, 0x4002507000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a0d00, 0x4002507000, 0x1000, 0x1000, 0x40, 0x1d, 0x40001457b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194420, 0x4002507000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023fc5a0, 0x4002507000, 0x1000, 0x1000, 0x4004494480, 0x200000003, 0x4004494480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40025007e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40025007e0, 0x40001e200a, 0x40001458f8, 0x412bc0, 0x40000820c0, 0x40001e2000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40025007e0, 0x0, 0x1, 0x4dd090, 0x40000820d8, 0x40001e2000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003f8c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003f8c30, 0x4004531000, 0x1000, 0x1000, 0x40001459b8, 0x0, 0x4000145988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011b440, 0x4004531000, 0x1000, 0x1000, 0x4000102900, 0x3, 0x4000145be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011b440, 0x4004531000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011b480, 0x4004531000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000145dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000145dd0, 0xa, 0x4004531201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000145dd0, 0x400453120a, 0x1, 0x4004526639, 0x0, 0x40045312e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000145dd0, 0x400450640a, 0x4004526639, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004484520, 0x40044a02a0, 0x17, 0x4000145f20, 0x400448e828, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4004484520, 0x113075a, 0x7, 0x400448e828, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40044844e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011cc98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011cc80, 0x40001e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011cc80, 0x40001e7000, 0x1000, 0x1000, 0x40, 0x1d, 0x400009b7b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010728, 0x40001e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000399440, 0x40001e7000, 0x1000, 0x1000, 0x4004494600, 0x200000003, 0x4004494600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400018e9c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400018e9c0, 0x40001e200a, 0x400009b8c8, 0x4d3b10, 0x40000820c0, 0x40001e2000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400018e9c0, 0x0, 0x1, 0x4dd090, 0x40000820d8, 0x40001e2000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400042db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400042db00, 0x400444b000, 0x1000, 0x1000, 0x400009b9b8, 0x10, 0x400009ba68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003dcdc0, 0x400444b000, 0x1000, 0x1000, 0x4004492898, 0x58, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003dcdc0, 0x400444b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003dce00, 0x400444b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bdd0, 0xa, 0x400444b201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bdd0, 0x400444b20a, 0x1, 0x400452663c, 0x0, 0x400444b263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bdd0, 0x400004690a, 0x400452663c, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004484520, 0x40044865a0, 0x24, 0x400009bf20, 0x400448e810, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4004484520, 0x113075a, 0x7, 0x400448e810, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40044844e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 33 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40003ddbc0, 0x4000112f50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f6500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40044931a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400449aea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40044b6600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 77 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d1d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40044b6700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40044b6700, 0x0, 0x400251fe98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045b51b8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045b51b8, 0x12ed9d0, 0x40045b51b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40044b6680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004484d00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 29 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4004484d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004470210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 32 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000219360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 31 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000219360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 99 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 167 [semacquire]: sync.runtime_Semacquire(0x4004ddc8c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ddc8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000d1600, 0x400010eab0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdf8d6969, 0x29, 0xffffdf8d69cb, 0x25, 0x40001ff8c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b27d20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdf8d6969, 0x29, 0xffffdf8d69cb, 0x25, 0x40001ff8c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b27d20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d53190, 0x4004d52e60, 0x400010eab0, 0x40023ee240, 0x40023f6080, 0x0, 0x3f, 0x40034c6da8, 0xffff8824e930, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000420700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004b27d20, 0xb, 0x4004b27d18, 0x6, 0x4004b27d30, 0xd, 0x4004b6ac20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000420700, 0x40003a0000, 0x4000205fb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000420700, 0x4004482ef0, 0x2, 0x2, 0x4004482f00, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 80 [select]: net/http.(*persistConn).readLoop(0x40044c2900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x40044c2900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f7440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011d900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011d900, 0x58ec48, 0x40034f5b60, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b69818, 0x4714f4, 0x40034f5b60, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b69818, 0x4004526c00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400011bf00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 102 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40003ddbc0, 0x400007f110, 0x400007f180, 0x4000112f50, 0x12e8f90, 0x40024a69a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b27034, 0x5, 0x400011bf00, 0x12eca70, 0x400011bdc0, 0x130d478, 0x40000e0620, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40044b4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40044b4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 103 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40024a69a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 104 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40024a69a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 105 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002528360, 0x4004ae2d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 106 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40023f6180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 107 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400045b220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 108 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400045b220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 109 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010a68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 110 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400045b220, 0x4004ae2d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 111 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c300, 0x1144559, 0x1b, 0x40000a1e38, 0x4004ae2ea0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b7b000, 0x4004ae2ea0, 0xf1f700, 0x400021db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a1f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4000115920, 0x4004b7b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 112 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 113 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40001c2f00, 0xd00007, 0x4004c8e9e8, 0x5, 0x4004b7b010, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d3a8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011db18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011db00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011db00, 0xffffb1053108, 0x30, 0x4004b71aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b69db8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b69db8, 0x4004b71aa0, 0x11ae500, 0x40044b4cc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044b4c40, 0x12ed9d0, 0x4004b69db8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044b4c40, 0x12ed9d0, 0x4004b69db8, 0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011dc18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011dc00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011dc00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b69dd0, 0xffffb1053108, 0xffff882ae648, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b69dd0, 0x665e1c, 0x0, 0xfe5e60, 0x40046a7dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b69de8, 0x4004b71b30, 0x11ae500, 0x40044b4da0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044b4d20, 0x12ec710, 0x4004b69de8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044b4d20, 0x12ec710, 0x4004b69de8, 0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d0f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6800, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6800, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae0d8, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044c2900, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044937a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044937a0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40046a7c18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40044937a0, 0x412bc0, 0x0, 0x0, 0x40046a7cc8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004b71b90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004b71b90, 0x4004b2f000, 0x200, 0x200, 0x4004b71b00, 0x30, 0x40046a7d78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081b40, 0x4004b2f000, 0x200, 0x200, 0x0, 0xffffb1053108, 0x40046a7d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081b40, 0x4004b2f000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081bc0, 0x4004b2f000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b2b760, 0x12ec700, 0x40046a7eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b2b760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b2b760, 0xee0d40, 0x40001c2ff0, 0x12db068, 0x4000081bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c8c750, 0x4004ca4000, 0x4004ae3020) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 163 [IO wait]: internal/poll.runtime_pollWait(0xffff88528028, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6980, 0x40001e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6980, 0x40001e0000, 0x1000, 0x1000, 0x4004540be0, 0x469b10, 0x4004494900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae130, 0x40001e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023fd560, 0x40001e0000, 0x1000, 0x1000, 0x4004540cf8, 0x67867c, 0x400014f800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004493bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004493bc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023fd560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 169 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 164 [select]: net/http.(*persistConn).writeLoop(0x40023fd560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40044a2f40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 186 [IO wait]: internal/poll.runtime_pollWait(0xffff88527f40, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6c00, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6c00, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae218, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044c2d80, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d84c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d84c60, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000129438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d84c60, 0x0, 0x4000129478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010ffb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010ffb0, 0x40001d7000, 0x200, 0x200, 0x4, 0x0, 0x4000129508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044a30c0, 0x40001d7000, 0x200, 0x200, 0x40001d7000, 0x200, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044a30c0, 0x40001d7000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044a3100, 0x40001d7000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d82b00, 0x0, 0x40001296b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d82b00, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d82b00, 0xee0d40, 0x4004490870, 0x12db068, 0x40044a3100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d80750, 0x4004d84f00, 0x40001ff2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 184 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6b00, 0x4004d98000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6b00, 0x4004d98000, 0x1000, 0x1000, 0x4004d9fbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae220, 0x4004d98000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044c2c60, 0x4004d98000, 0x1000, 0x1000, 0x4004d9fcf8, 0x67867c, 0x40001ff260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d84d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d84d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40044c2c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 175 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40044993f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 176 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40044993f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 177 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40044993f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 178 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 179 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c300, 0x1144559, 0x1b, 0x4000147e38, 0x40001fe540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004483bd0, 0x40001fe540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000147f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004493f20, 0x4004483bd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 185 [select]: net/http.(*persistConn).writeLoop(0x40044c2c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 182 [select]: net/http.(*persistConn).readLoop(0x40044c2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 183 [select]: net/http.(*persistConn).writeLoop(0x40044c2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 187 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044a2f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 188 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 189 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 190 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 191 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 192 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40044bbe00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 193 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40044bbe00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 194 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 195 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 196 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 197 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:44.344+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:44.345+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:44.352+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:44.358+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:44.358+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:44.358+00:00 [Info] Setting log level to Info 2021-08-16T13:47:44.358+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:44.361+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:44 audit: created new audit service 2021-08-16T13:47:44.369+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:44.369+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:44.370+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:44.370+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:44.370+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:44.380+00:00 [Info] security context initialized 2021-08-16T13:47:44.380+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:44.381+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:44.381+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:44.381+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:44.381+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:44.394+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:44.395+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:44.407+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:44.425+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:44.425+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:44.427+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:44.427+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:44.429+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:44.429+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:44.430+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:44.430+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:44.480+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:44.480+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 204 us 2021-08-16T13:47:44.481+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:44.482+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:44.483+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:44.486+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:44.486+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:44.486+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:44.486+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:44.486+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:44.486+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.486+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.488+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:44.488+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:44.488+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:44.488+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.488+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:44.488+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:44.488+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:44.488+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:44.488+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:44.488+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:44.488+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:44.494+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.204778ms 2021-08-16T13:47:44.494+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:44.495+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.495+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:44.506+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:44.506+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:44.506+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:44.506+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:44.506+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40001cdd5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40001cdd5f pc=0xbd8ea4] goroutine 189 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004f5c9f0 sp=0x4004f5c9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004f5ca30 sp=0x4004f5c9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400444f380, 0x4004e16c00, 0x11a9890, 0x4004f5cb40, 0x400444f398, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004f5cab0 sp=0x4004f5ca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400444f380, 0x4004e16c00, 0x11a9890, 0x4004f5cb40, 0x400444f398, 0x3f, 0x4004f5cb78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004f5caf0 sp=0x4004f5cab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000396640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004f5cda0 sp=0x4004f5caf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000396640, 0x40013a5180, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004f5ce60 sp=0x4004f5cda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e64a00, 0x4004e64a10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004f5d200 sp=0x4004f5ce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e64a00, 0x4004e64a10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004f5d8f0 sp=0x4004f5d200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004f4b000, 0x40013d9500, 0x4002491080, 0x4004e649e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004f5dfb0 sp=0x4004f5d8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004f5dfb0 sp=0x4004f5dfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000193dd0, 0x4, 0x1136a87, 0xf, 0x4000191e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400012a678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400012a690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5ca388, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188880, 0x40001bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188880, 0x40001bd000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001385a0, 0x40001bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400013ac60, 0x40001c2001, 0xdff, 0xdff, 0x400011d8e8, 0x6902d8, 0x400018d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400012a6d8, 0x40001c2001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001669a0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001669a0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001669a0, 0xf5a960, 0x400013ad40, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400013ad20, 0x4000142780, 0x400018db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000100820, 0x12f8f98, 0x400013ad20, 0x4000135407, 0x16, 0x4000147440, 0x400011dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000100820, 0x12f8f98, 0x400013ad20, 0x4000100820, 0x4000134c18, 0x4000134c20, 0x4000188900, 0x40001426a0, 0xee0980, 0x4000147440, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000100820, 0x12f8f98, 0x400013ad20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001923c0, 0x400011df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011bf78, 0x40001923c0, 0x12db8a8, 0x400012a6a8, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001923c0, 0x400013d2c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001923c0, 0x400013d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 24 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 25 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 26 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 27 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 28 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 5 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 29 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003d47a0, 0x4, 0x40003d4668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003d47a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400013a000, 0x40003d47a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000138000, 0x40003d47a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 13 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001ae8d0, 0x12c18bc, 0x1, 0x400011de58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003d56c8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400011dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 11 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 12 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 37 [select]: net/http.(*persistConn).writeLoop(0x400032e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 34 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5ca1b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188180, 0x40001c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188180, 0x40001c3000, 0x1000, 0x1000, 0x40000eebe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001382e8, 0x40001c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x40001c3000, 0x1000, 0x1000, 0x40000eecf8, 0x67867c, 0x4000102660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003596e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003596e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 35 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5ca2a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e6118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e6100, 0x40000f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e6100, 0x40000f2000, 0x1000, 0x1000, 0x40000eaa88, 0x68f7a4, 0x40003ee050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010038, 0x40000f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x40000f2000, 0x1000, 0x1000, 0xf12180, 0x40000942c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000092840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000092840, 0xa, 0x1008880, 0x4000094280, 0x199, 0x1000, 0x40000eab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000092840, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000114d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000114d50, 0x40003f4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000158100, 0x40003f4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000158100, 0x40003f4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000158580, 0x40003f4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40003ee000, 0x0, 0x40000eaeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40003ee000, 0x0, 0x0, 0x40000eaf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40003ee000, 0xee0d40, 0x4000094280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40003da000, 0x400013a720, 0x40003ec000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 36 [select]: net/http.(*persistConn).readLoop(0x400032e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 32 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032ce80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 82 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 84 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c98a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e6198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e6180, 0x40044f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e6180, 0x40044f3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010090, 0x40044f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400032e900, 0x40044f3000, 0x1000, 0x1000, 0x400009ea80, 0x200000003, 0x400009ea80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000093020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000093020, 0x40000fc00a, 0x40000a78c8, 0x4d3b10, 0x400013a060, 0x40000fc000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000093020, 0x0, 0x1, 0x2f, 0x40000a78d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400253e420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400253e420, 0x40044f6000, 0x1000, 0x1000, 0x40000a79b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400032c3c0, 0x40044f6000, 0x1000, 0x1000, 0x4000092958, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400032c3c0, 0x40044f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400032c480, 0x40044f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a7dd0, 0xa, 0x40044f6201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a7dd0, 0x40044f620a, 0x1, 0x400253cf19, 0x0, 0x40044f62e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a7dd0, 0x400253a40a, 0x400253cf19, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b0860, 0x400003a330, 0x17, 0x40000a7f20, 0x40013fa378, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000b0860, 0x113075a, 0x7, 0x40013fa378, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000b0820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 81 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032cd80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 102 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 167 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40000812c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000e98c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 85 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000159e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 86 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40044125a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40044224b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4004442400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9b60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004442518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004442500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004442500, 0x0, 0x4004410e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004420690, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004420690, 0x12ed9d0, 0x4004420690, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4004442480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004418680, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004418680, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4004418680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004422f90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000178070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c96d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188280, 0x40044fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188280, 0x40044fa000, 0x1000, 0x1000, 0x400256d688, 0x69f178, 0x400256d6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001380e0, 0x40044fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023b6120, 0x40044fa000, 0x1000, 0x1000, 0x5, 0x400241c000, 0x400256d7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023b2660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023b2660, 0x68f90a, 0x40000e86e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023b2660, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400253eb10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400253eb10, 0x40000b2000, 0x1000, 0x1000, 0x400256d998, 0x44e884, 0xffff97271a68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400032cac0, 0x40000b2000, 0x1000, 0x1000, 0x4000092958, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400032cac0, 0x40000b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400032cb00, 0x40000b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400256dd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400256dd80, 0xa, 0x40000b2201, 0x1, 0xffff6c026c98, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400256dd80, 0x40000b220a, 0x1, 0x40023b8820, 0x0, 0x40000b2259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400256dd80, 0x40000e3c0a, 0x40023b8820, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b0860, 0x4002565e60, 0x1f, 0x400256dee0, 0x4002566450, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000b0860, 0x113075a, 0x7, 0x400253cf30, 0xd, 0x4002566450, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000b0820, 0x400253cf30, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 69 [select]: net/http.(*persistConn).readLoop(0x40001470e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 70 [select]: net/http.(*persistConn).writeLoop(0x40001470e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 57 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9990, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002486b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002486b00, 0x40044e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002486b00, 0x40044e7000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003fe1d8, 0x40044e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001470e0, 0x40044e7000, 0x1000, 0x1000, 0x400009ef00, 0x200000003, 0x400009ef00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002485080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002485080, 0x40000fc00a, 0x40000a58c8, 0x4d3b10, 0x400013a060, 0x40000fc000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002485080, 0x0, 0x1, 0x1, 0x40000a58d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400007e030) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400007e030, 0x40001c7000, 0x1000, 0x1000, 0x40000a59b8, 0x0, 0x40000a5988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024c8800, 0x40001c7000, 0x1000, 0x1000, 0x4001396480, 0x3, 0x40000a5be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024c8800, 0x40001c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024c8840, 0x40001c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a5dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a5dd0, 0xa, 0x40001c7201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a5dd0, 0x40001c720a, 0x1, 0x40044daa8b, 0x0, 0x40001c7263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a5dd0, 0x400004600a, 0x40044daa8b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b0860, 0x4000042360, 0x24, 0x40000a5f20, 0x40013fa360, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000b0860, 0x113075a, 0x7, 0x40013fa360, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000b0820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 60 [select]: net/http.(*persistConn).readLoop(0x400032e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 61 [select]: net/http.(*persistConn).writeLoop(0x400032e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 76 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024c89c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [select]: net/http.(*persistConn).readLoop(0x40023b6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 75 [select]: net/http.(*persistConn).writeLoop(0x40023b6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c97c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000189998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000189980, 0x40045f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000189980, 0x40045f7000, 0x1000, 0x1000, 0x400257abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000138868, 0x40045f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023b7680, 0x40045f7000, 0x1000, 0x1000, 0x400257acf8, 0x67867c, 0x4004b54480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045f8420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045f8420, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023b7680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [semacquire]: sync.runtime_Semacquire(0x4004f4b008) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004f4b000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4002491080, 0x4000384ed0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffed57c969, 0x29, 0xffffed57c9cb, 0x25, 0x4004e614a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b95fa0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffed57c969, 0x29, 0xffffed57c9cb, 0x25, 0x4004e614a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b95fa0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004f5d190, 0x4004f5ce60, 0x4000384ed0, 0x400234cb40, 0x400032ce80, 0x0, 0xbf, 0x40000a0da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000aa000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004b95fa0, 0xb, 0x4004b95f98, 0x6, 0x4004b95fb0, 0xd, 0x4004bb3600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000aa000, 0x4000350000, 0x40000a8030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000aa000, 0x4004416530, 0x2, 0x2, 0x4004416540, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x40023b7680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9a78, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004bbc218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004bbc200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004bbc200, 0x12d9d48, 0x400032dc40, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb5278, 0x0, 0x40003ec1e0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb5278, 0x400253da00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004be00c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 104 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400442f380, 0x400019d1f0, 0x400019d260, 0x4004417fa0, 0x12e8f90, 0x4004440700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b95014, 0x5, 0x4004be00c0, 0x12eca70, 0x4000159c80, 0x130d478, 0x40000e4310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4004440d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4004440d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 95 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400441cbe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 96 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400441cbe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 97 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400442f380, 0x4004417fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4004440700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 115 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4004440700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40044e0bd0, 0x400456c660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40024c8980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400441cc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400441cc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400443a318) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 121 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400441cc80, 0x400456c6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 122 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001ae8d0, 0x1144559, 0x1b, 0x4004779e38, 0x400456c7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004760ac0, 0x400456c7e0, 0xf1f700, 0x40003ce150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004779f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400476e4e0, 0x4004760ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400444a8c0, 0xd00007, 0x400456b958, 0x5, 0x4004760ad0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 125 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c95f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004442b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004442b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004442b80, 0xffff97271f18, 0x30, 0x400472f020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400472c7c8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400472c7c8, 0x400472f020, 0x11ae500, 0x400235a860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400235a7e0, 0x12ed9d0, 0x400472c7c8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400235a7e0, 0x12ed9d0, 0x400472c7c8, 0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 126 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9508, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004442c98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004442c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004442c80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400472c7e0, 0xffff97271f18, 0xffff575b0820, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400472c7e0, 0x665e1c, 0x0, 0xfe5e60, 0x4004759dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400472c7f8, 0x400472f0b0, 0x11ae500, 0x400235a940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400235a8c0, 0x12ec710, 0x400472c7f8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400235a8c0, 0x12ec710, 0x400472c7f8, 0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9c48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e6c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e6c00, 0x40001bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e6c00, 0x40001bf000, 0x1000, 0x1000, 0x4004bd9ad8, 0x412bc0, 0x4004bd9a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001387f0, 0x40001bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400032e240, 0x40001bf000, 0x1000, 0x1000, 0xffff572ef328, 0x0, 0x14) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003599e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003599e0, 0xa, 0x40001c0123, 0x11320c7, 0x9, 0x1, 0x4004759c9b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003599e0, 0xd4fa28, 0x4000132200, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400472fe60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400472fe60, 0x4004d18600, 0x200, 0x200, 0x400456bce0, 0x400456bce0, 0x400442a810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400442fb40, 0x4004d18600, 0x200, 0x200, 0x0, 0xffff97271f18, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400442fb40, 0x4004d18600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400442fb80, 0x4004d18600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004571ce0, 0x12ec700, 0x4004759eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004571ce0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004571ce0, 0xee0d40, 0x400444af50, 0x12db068, 0x400442fb80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4002345950, 0x400476f260, 0x400456d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff57309d80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004bbc418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004bbc400, 0x4004d80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004bbc400, 0x4004d80000, 0x1000, 0x1000, 0x4004bdabe0, 0x469b10, 0x400476d080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000138c08, 0x4004d80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044537a0, 0x4004d80000, 0x1000, 0x1000, 0x4004bdacf8, 0x67867c, 0x4004b557a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045f9200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045f9200, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40044537a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x40044537a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000e98c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40000e98c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 172 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001ae8d0, 0x1144559, 0x1b, 0x40000a3e38, 0x4004e600c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004e64050, 0x4004e600c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a3f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004e42300, 0x4004e64050) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 177 [IO wait]: internal/poll.runtime_pollWait(0xffff57309e68, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e7718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e7700, 0x40013e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e7700, 0x40013e3000, 0x1000, 0x1000, 0x4002512a48, 0x4000107b00, 0x4000107b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102b8, 0x40013e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400032f9e0, 0x40013e3000, 0x1000, 0x1000, 0x0, 0x4000150dc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e42ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e42ba0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400440dc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e42ba0, 0x0, 0x400440dc78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e4ae10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e4ae10, 0x4004e4c600, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081400, 0x4004e4c600, 0x200, 0x200, 0x4000035c07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081400, 0x4004e4c600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081440, 0x4004e4c600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e49760, 0x0, 0x400440deb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e49760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e49760, 0xee0d40, 0x4000344aa0, 0x12db068, 0x4000081440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e623f0, 0x4004e42d20, 0x4004e60c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 175 [select]: net/http.(*persistConn).readLoop(0x400032f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 176 [select]: net/http.(*persistConn).writeLoop(0x400032f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 178 [runnable]: net.IP.IsLoopback(0x0, 0x0, 0x0, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ip.go:121 +0x130 github.com/couchbase/cbauth/cbauthimpl.matchHost(0x4000154360, 0x2e, 0x40001352e8, 0x6, 0x4000136780, 0x20, 0x40001640e0, 0x18, 0x1c, 0x1, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:114 +0x90 github.com/couchbase/cbauth/cbauthimpl.getMemcachedCreds(0x4000154360, 0x2e, 0x40001352e8, 0x6, 0x4000136780, 0x20, 0x40001640e0, 0x18, 0x1c, 0x1, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:129 +0x4c github.com/couchbase/cbauth/cbauthimpl.GetCreds(0x400013d2c0, 0x4004c32c07, 0x9, 0x1f9b, 0x1f9b, 0x0, 0x0, 0xe, 0x0, 0x4004c32c00, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:791 +0xc0 github.com/couchbase/cbauth.(*authImpl).GetHTTPServiceAuth(0x4000138530, 0x4004c32c07, 0xe, 0x4004ea05a0, 0x15, 0x0, 0x400256f268, 0x7ff1dc, 0x6) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauth.go:188 +0x68 github.com/couchbase/cbauth.SetRequestAuthVia.func1(0x1303cf8, 0x4000138530, 0x400256f248, 0x4004c32c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/convenience.go:31 +0x4c github.com/couchbase/cbauth.WithAuthenticator(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:136 github.com/couchbase/cbauth.SetRequestAuthVia(0x4004ea4300, 0x0, 0x0, 0x3, 0x4004c32c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/convenience.go:30 +0x54 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004c32bd0, 0x1b, 0x400256f580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:537 +0x13c github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004ea0480, 0x112f905, 0x6, 0x12db908, 0x4000047080, 0xee40c0, 0x4002487498, 0x400256f580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004478280, 0x46, 0x12db908, 0x4000047080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004478280, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400256fd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40001ccfc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40001ccfc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 183 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004e47100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004e47100, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 187 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 188 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:49.667+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:49.668+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:49.681+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:49.683+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:49.683+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:49.683+00:00 [Info] Setting log level to Info 2021-08-16T13:47:49.683+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:49.685+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:49 audit: created new audit service 2021-08-16T13:47:49.694+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:49.694+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:49.695+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:49.695+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:49.695+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:49.707+00:00 [Info] security context initialized 2021-08-16T13:47:49.707+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:49.707+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:49.708+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:49.708+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:49.708+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:49.721+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:49.723+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:49.725+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:49.747+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:49.747+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:49.747+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:49.748+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:49.749+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:49.750+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:49.750+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:49.752+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:49.800+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:47:49.800+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:49.799+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:49.801+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:49.802+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:49.806+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:49.806+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:49.806+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:49.806+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:49.806+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:49.806+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.806+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.808+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:49.808+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:49.808+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:49.808+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.808+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:49.808+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:49.808+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:49.808+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:49.808+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:49.808+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:49.808+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:49.813+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.023069ms 2021-08-16T13:47:49.814+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:49.815+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:49.816+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:49.826+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:49.826+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:49.826+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:49.827+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:47:49.827+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:49.827+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d unexpected fault address 0x400476345f fatal error: fault 2021-08-16T13:47:49.827+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x400476345f pc=0xbd8ea4] goroutine 195 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d229f0 sp=0x4004d229c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d22a30 sp=0x4004d229f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004770b60, 0x4004701200, 0x11a9890, 0x4004d22b40, 0x4004770b78, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d22ab0 sp=0x4004d22a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004770b60, 0x4004701200, 0x11a9890, 0x4004d22b40, 0x4004770b78, 0xffff85fb5a90, 0x4004d22b78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d22af0 sp=0x4004d22ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032dc0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d22da0 sp=0x4004d22af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032dc0, 0x40001a5b20, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d22e60 sp=0x4004d22da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004731f10, 0x4004731f20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d23200 sp=0x4004d22e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004731f10, 0x4004731f20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d238f0 sp=0x4004d23200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dba4e0, 0x4004dc6000, 0x4000171b80, 0x4004731ef0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d23fb0 sp=0x4004d238f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d23fb0 sp=0x4004d23fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400019b5c0, 0x4, 0x1136a87, 0xf, 0x4000199e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000130678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000130690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff85e54438, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190880, 0x40001c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190880, 0x40001c5000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e5a0, 0x40001c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000140c60, 0x40000a6001, 0xdff, 0xdff, 0x40001238e8, 0x6902d8, 0x4000195948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001306d8, 0x40000a6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000a0000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a0000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a0000, 0xf5a960, 0x40000824a0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082480, 0x4000046140, 0x4000195b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c230, 0x12f8f98, 0x4000082480, 0x40000368a7, 0x16, 0x400000c240, 0x4000123c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c230, 0x12f8f98, 0x4000082480, 0x400008c230, 0x40000360b0, 0x40000360c0, 0x400009e000, 0x4000046060, 0xee0980, 0x400000c240, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c230, 0x12f8f98, 0x4000082480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400019a3c0, 0x4000123f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000121f78, 0x400019a3c0, 0x12db8a8, 0x40001306a8, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400019a3c0, 0x40001432c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400019a3c0, 0x40001432c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 5 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003c87a0, 0x4, 0x40003c8668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003c87a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x40003c87a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x40003c87a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53c10, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004422318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004422300, 0x4004559000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004422300, 0x4004559000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400441e0b0, 0x4004559000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400442a900, 0x4004559000, 0x1000, 0x1000, 0x40000f8000, 0x200000003, 0x40000f8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000351ec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000351ec0, 0x40000ac00a, 0x40023578c8, 0x4d3b10, 0x4000140060, 0x40000ac000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000351ec0, 0x0, 0x1, 0x14, 0x40023578d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013d0060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013d0060, 0x40003ee000, 0x1000, 0x1000, 0x40023579b8, 0x0, 0x4002357988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002476000, 0x40003ee000, 0x1000, 0x1000, 0x4001396480, 0x3, 0x4002357be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002476000, 0x40003ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002476040, 0x40003ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002357dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002357dd0, 0xa, 0x40003ee201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002357dd0, 0x40003ee20a, 0x1, 0x40045548f9, 0x0, 0x40003ee2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002357dd0, 0x400009c50a, 0x40045548f9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400442ec70, 0x4004414630, 0x17, 0x4002357f20, 0x400454df68, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400442ec70, 0x113075a, 0x7, 0x400454df68, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400442ec30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 12 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 13 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 34 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b6440, 0x12c18bc, 0x1, 0x40013b5e58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400001dec8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40013b5fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40000801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 35 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40000801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffff85e54268, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190080, 0x4000096000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190080, 0x4000096000, 0x1000, 0x1000, 0x40013a4be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e310, 0x4000096000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400033ea20, 0x4000096000, 0x1000, 0x1000, 0x40013a4cf8, 0x67867c, 0x400001e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082cc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400033ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400033ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [IO wait]: internal/poll.runtime_pollWait(0xffff85e54350, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009e218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009e200, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009e200, 0x40000a7000, 0x1000, 0x1000, 0x40013a8a88, 0x68f7a4, 0x40000a05d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010198, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x40000a7000, 0x1000, 0x1000, 0xf12180, 0x400008c3b0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082a20, 0xa, 0x1008880, 0x400008c370, 0x199, 0x1000, 0x40013a8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082a20, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400011b1d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400011b1d0, 0x40013bc000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000160580, 0x40013bc000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000160580, 0x40013bc000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161580, 0x40013bc000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000a0580, 0x0, 0x40013a8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a0580, 0x0, 0x0, 0x40013a8f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a0580, 0xee0d40, 0x400008c370, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000a8090, 0x4000140900, 0x4000038ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 123 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400009abe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001fae80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 51 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000161c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 70 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001fb9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 215 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdede0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009ed98, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009ed80, 0x4004e10971, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009ed80, 0x4004e10971, 0x1, 0x1, 0x40023d4601, 0x0, 0x40003586e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22e8, 0x4004e10971, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004e10960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 48 [select]: net/http.(*persistConn).readLoop(0x40024ba480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 49 [select]: net/http.(*persistConn).writeLoop(0x40024ba480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53b28, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191580, 0x40024f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191580, 0x40024f6000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e950, 0x40024f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024ba480, 0x40024f6000, 0x1000, 0x1000, 0x4000102f00, 0x200000003, 0x4000102f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002498e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002498e40, 0x40000ac00a, 0x40023558c8, 0x4d3b10, 0x4000140060, 0x40000ac000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002498e40, 0x0, 0x1, 0x0, 0x4000140078, 0x40000ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400033a5d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400033a5d0, 0x40044e4000, 0x1000, 0x1000, 0x40023559b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003de200, 0x40044e4000, 0x1000, 0x1000, 0x4000351a98, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003de200, 0x40044e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003de240, 0x40044e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002355dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002355dd0, 0xa, 0x40044e4201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002355dd0, 0x40044e420a, 0x1, 0x40044b157b, 0x0, 0x40044e4263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002355dd0, 0x40003e210a, 0x40044b157b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400442ec70, 0x4004410a80, 0x24, 0x4002355f20, 0x400454df50, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400442ec70, 0x113075a, 0x7, 0x400454df50, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400442ec30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 85 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000812c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 83 [select]: net/http.(*persistConn).readLoop(0x400442a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 84 [select]: net/http.(*persistConn).writeLoop(0x400442a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4004566120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000391710) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4004422380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53a40, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004422498, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004422480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004422480, 0x0, 0x40003ca698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004568228, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004568228, 0x12ed9d0, 0x4004568228, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4004422400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400442f040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400442f040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400442f040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004570210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000f6070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53870, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002466298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002466280, 0x40013bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002466280, 0x40013bd000, 0x1000, 0x1000, 0x40013af688, 0x69f178, 0x40013af6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010068, 0x40013bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e5a0, 0x40013bd000, 0x1000, 0x1000, 0x5, 0x4004460500, 0x40013af7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004566ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004566ae0, 0x68f90a, 0x40003d4a50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004566ae0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400033aed0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400033aed0, 0x40044e5000, 0x1000, 0x1000, 0x40013af998, 0x44e884, 0x40013af948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003de900, 0x40044e5000, 0x1000, 0x1000, 0x4000351a98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003de900, 0x40044e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003de940, 0x40044e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40013afd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013afd80, 0xa, 0x40044e5201, 0x1, 0xffff85fbe658, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40013afd80, 0x40044e520a, 0x1, 0x4000037260, 0x0, 0x40044e5259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40013afd80, 0x40024f3a0a, 0x4000037260, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400442ec70, 0x400454e9c0, 0x1f, 0x40013afee0, 0x4004568540, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400442ec70, 0x113075a, 0x7, 0x4004554910, 0xd, 0x4004568540, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400442ec30, 0x4004554910, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 58 [select]: net/http.(*persistConn).readLoop(0x400014e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).writeLoop(0x400014e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 106 [select]: net/http.(*persistConn).readLoop(0x40024ba5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53788, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191b80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb6420, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb6420, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4002476e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 95 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003df000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x40024ba5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 211 [select]: net/http.(*persistConn).writeLoop(0x40024bb8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff85e536a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191a80, 0x40045ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191a80, 0x40045ca000, 0x1000, 0x1000, 0x4004576a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f2048, 0x40045ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024ba5a0, 0x40045ca000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x400008c460) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002499740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002499740, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x400001a438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002499740, 0x0, 0x400001a478, 0x655784, 0x400001a4a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045b1c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045b1c20, 0x40045fae00, 0x200, 0x200, 0x4, 0x0, 0x400001a508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080f00, 0x40045fae00, 0x200, 0x200, 0x40000395c7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080f00, 0x40045fae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080f40, 0x40045fae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40023d4580, 0x400456f800, 0x400001a6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40023d4580, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40023d4580, 0xee0d40, 0x4000358690, 0x12db068, 0x4000080f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400239a750, 0x4000083140, 0x40045f6b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 97 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004734140, 0x40013b8a10, 0x40013b8a80, 0x4004730240, 0x12e8f90, 0x400240e460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40000e1dd4, 0x5, 0x4002476e80, 0x12eca70, 0x4002476d40, 0x130d478, 0x40000f60e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40023868c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40023868c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 121 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53958, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002467498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002467480, 0x400241b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002467480, 0x400241b000, 0x1000, 0x1000, 0x4004573be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010350, 0x400241b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400033e240, 0x400241b000, 0x1000, 0x1000, 0x4004573cf8, 0x67867c, 0x40013eea80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013f06c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40013f06c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400033e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 122 [select]: net/http.(*persistConn).writeLoop(0x400033e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 124 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400009abe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 125 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004734140, 0x4004730240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 126 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400240e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 127 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400240e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 128 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40000fbc20, 0x40013efc20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40001fae40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400009ac80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 131 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400009ac80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010550) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 133 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400009ac80, 0x40013efc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 134 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b6440, 0x1144559, 0x1b, 0x4002353e38, 0x40013efda0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004730d60, 0x40013efda0, 0xf1f700, 0x40001b7cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002353f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40013f19e0, 0x4004730d60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 135 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d180, 0xd00007, 0x4004717968, 0x5, 0x4004730d70, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 137 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53cf8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002467c18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002467c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002467c00, 0x0, 0x40013ead88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476c1f8, 0x611a6c85, 0x40013eadb8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476c1f8, 0x40013eae20, 0x40013eae28, 0x18, 0x4002348f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400447cc40, 0x12ed9d0, 0x400476c1f8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400447cc40, 0x12ed9d0, 0x400476c1f8, 0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 138 [IO wait]: internal/poll.runtime_pollWait(0xffff85e535b8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002467d18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002467d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002467d00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476c210, 0xffff85fb5a68, 0xffff4dfc26f8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476c210, 0x665e1c, 0x0, 0xfe5e60, 0x40045b65c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400476c228, 0x400472b5c0, 0x11ae500, 0x400447cda0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400447cd20, 0x12ec710, 0x400476c228, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400447cd20, 0x12ec710, 0x400476c228, 0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004cf08f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 145 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdf268, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d72098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d72080, 0x4004d3d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d72080, 0x4004d3d000, 0x1000, 0x1000, 0x40013e4be0, 0x469b10, 0x4002349500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010600, 0x4004d3d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400476e480, 0x4004d3d000, 0x1000, 0x1000, 0x40013e4cf8, 0x67867c, 0x4004cea7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cf27e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cf27e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400476e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 163 [semacquire]: sync.runtime_Semacquire(0x4004dba4e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dba4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000171b80, 0x4000098300, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffda80d969, 0x29, 0xffffda80d9cb, 0x25, 0x4004cebce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd8ac0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffda80d969, 0x29, 0xffffda80d9cb, 0x25, 0x4004cebce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd8ac0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d23190, 0x4004d22e60, 0x4000098300, 0x4000153440, 0x4000161c80, 0x0, 0x7f, 0x40044e8da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000c0380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bd8ac0, 0xb, 0x4004bd8ab8, 0x6, 0x4004bd8ad0, 0xd, 0x4004bb2e80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000c0380, 0x4000350000, 0x4000098840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000c0380, 0x4004550bd0, 0x2, 0x2, 0x4004550be0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 146 [select]: net/http.(*persistConn).writeLoop(0x400476e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 168 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4002477240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004cf08f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004cf08f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 157 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b6440, 0x1144559, 0x1b, 0x40013b3e38, 0x4004bbc840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004731d70, 0x4004bbc840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40013b3f80, 0x12edba8, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002465560, 0x4004731d70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 194 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x40045bb7b0, 0x0, 0x400014a9b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 160 [select]: net/http.(*persistConn).readLoop(0x400476ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 161 [select]: net/http.(*persistConn).writeLoop(0x400476ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 169 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdf180, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d72318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d72300, 0x4004dbc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d72300, 0x4004dbc000, 0x1000, 0x1000, 0x40013a2a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010748, 0x4004dbc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400476ea20, 0x4004dbc000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x400008c460) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cf3920) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cf3920, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cf3920, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bb95c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bb95c0, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002477300, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002477300, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002477340, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b7ba20, 0x0, 0x40045bd6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b7ba20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b7ba20, 0xee0d40, 0x400014ae60, 0x12db068, 0x4002477340) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bbab40, 0x40024657a0, 0x4004bbce40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 170 [select]: net/http.(*persistConn).roundTrip(0x400033e240, 0x4002477700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004eaa600, 0xf8, 0x10a8660, 0x4004cf8e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004eaa600, 0x1b6e900, 0xc03eb91f7144c414, 0x1bfa71bc29) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004eaa500, 0x12db048, 0x1b6e900, 0xc03eb91f7144c414, 0x1bfa71bc29, 0x1be4a80, 0x0, 0x400013e9b0, 0x4004bb9bf0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004bb9bc0, 0x4004eaa500, 0xc03eb91f7144c414, 0x1bfa71bc29, 0x1be4a80, 0x400013e9b0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004bb9bc0, 0x4004eaa500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4000043cb0, 0x1b, 0x4004cf9580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004bbb290, 0x112f905, 0x6, 0x12db908, 0x4000047000, 0xee40c0, 0x4004eae398, 0x4004cf9580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004bc0200, 0x46, 0x12db908, 0x4000047000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004bc0200, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cf9d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000ab200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000ab200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4002477280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 171 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 173 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 174 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 175 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 176 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004eaa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 214 [select]: net/http.(*persistConn).roundTrip(0x4004e4e120, 0x4000081c00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e46600, 0xf8, 0x10a8660, 0x4004cf6588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e46600, 0x1b6e900, 0xc03eb91f7155f8ca, 0x1bfa82f0df) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004e46500, 0x12db048, 0x1b6e900, 0xc03eb91f7155f8ca, 0x1bfa82f0df, 0x1be4a80, 0x0, 0x40003f22f8, 0x4004e10bd0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e10ba0, 0x4004e46500, 0xc03eb91f7155f8ca, 0x1bfa82f0df, 0x1be4a80, 0x40003f22f8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e10ba0, 0x4004e46500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e48360, 0x1b, 0x4004cf6cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x400239b680, 0x112f905, 0x6, 0x12db908, 0x40001491a0, 0xee40c0, 0x400009ee98, 0x4004cf6cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023dc100, 0x46, 0x12db908, 0x40001491a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023dc100, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004e10ae0, 0xffff85e36658) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e694b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x400038afc0, 0x400038afc0, 0x40023dc100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffda80d94e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004e550c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4000081bc0, 0x4000358a00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004730c60, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4004734440, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x400447cc40, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x40001bda40, 0x12f86a0, 0x4000081a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 178 [select]: net/http.(*persistConn).roundTrip(0x40024bb8c0, 0x4002477740, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004eaa900, 0xf8, 0x10a8660, 0x40044e8dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004eaa900, 0x1b6e900, 0xc03eb91f714e585e, 0x1bfa7b506a) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004eaa800, 0x12db048, 0x1b6e900, 0xc03eb91f714e585e, 0x1bfa7b506a, 0x1be4a80, 0x0, 0x400013e9d8, 0x4004ec8060, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ec8030, 0x4004eaa800, 0xc03eb91f714e585e, 0x1bfa7b506a, 0x1be4a80, 0x400013e9d8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ec8030, 0x4004eaa800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4000043e30, 0x1b, 0x40044e9540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004bbb680, 0x112f905, 0x6, 0x12db908, 0x4000047080, 0xee40c0, 0x4004eae498, 0x40044e9540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004bc0400, 0x46, 0x12db908, 0x4000047080, 0x40013eb598, 0x81ee08, 0x40013eb748, 0x95dcbc, 0x400003e180, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004bc0400, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40044e9d00, 0x4004bb9f50, 0xffff85fc00e8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000ab680, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000ab680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 179 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 180 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 213 [select]: net/http.(*persistConn).writeLoop(0x40024bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 210 [runnable]: syscall.Syscall(0x3f, 0x1c, 0x4004e51000, 0x1000, 0x502, 0x4004e51000, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x1c, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x12dbfe8, 0x1b7f708) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4004eae580, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 net.(*netFD).Read(0x4004eae580, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22d8, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024bb8c0, 0x4004e51000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083bc0, 0x1, 0x40045f7aa0, 0x43d584, 0x0, 0x0, 0x400477e568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40024bb8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 183 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdeec8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004eae798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004eae780, 0x4004e50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004eae780, 0x4004e50000, 0x1000, 0x1000, 0x0, 0x4004e27000, 0x4004edfa38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22d0, 0x4004e50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x40000838c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000838c0, 0x40a, 0x4004e50000, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x40000838c0, 0x1000, 0xffff5c0673f8, 0x1, 0x4004edfbc8, 0x412ff8, 0x400239b440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004edfcf0, 0x400239b440, 0x4004edfc48, 0xffff5c0673f8, 0x2c8, 0xffff85fb5f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x40000838c0, 0x4004e46300, 0x1000, 0x4004e50000, 0x40003f22d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004ec8360, 0x4004edff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004edff10, 0x4004ec8360, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004eaa700, 0x0, 0x0, 0x4004572f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004eaa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 184 [select]: net/http.(*persistConn).roundTrip(0x40024bbb00, 0x4002477780, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004eaab00, 0xf8, 0x10a8660, 0x4004572798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004eaab00, 0x1b6e900, 0xc03eba2d71501b1a, 0x1176f81432f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004eaaa00, 0x12db048, 0x1b6e900, 0xc03eba2d71501b1a, 0x1176f81432f, 0x1be4a80, 0x0, 0x400013e9f0, 0x4004ec81e0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ec81b0, 0x4004eaaa00, 0xc03eba2d71501b1a, 0x1176f81432f, 0x1be4a80, 0x400013e9f0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ec81b0, 0x4004eaaa00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004ea7520, 0x1f, 0x4004572d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004ea7520, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004eba060, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004eaa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 212 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdefb0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004eae698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004eae680, 0x4004e53000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004eae680, 0x4004e53000, 0x1000, 0x1000, 0x40003ca3e8, 0x7ff1a8, 0x40003ca4b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22e0, 0x4004e53000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024bbb00, 0x4004e53000, 0x1000, 0x1000, 0x60, 0x0, 0x4004ea7501) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083c80, 0x1, 0x40045f7bc0, 0x43d584, 0x0, 0xd8c91c, 0x40003ca568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40024bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 218 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdecf8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009ef98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009ef80, 0x4004e5f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009ef80, 0x4004e5f000, 0x1000, 0x1000, 0x40003ca3e8, 0xffff4dfe37b0, 0x88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f2308, 0x4004e5f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e4e120, 0x4004e5f000, 0x1000, 0x1000, 0x60, 0x0, 0x4004ea7501) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e6a1e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e6a1e0, 0x1, 0x4004e6e000, 0x43d584, 0x0, 0xd8c91c, 0x40003ca568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e4e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 219 [select]: net/http.(*persistConn).writeLoop(0x4004e4e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:47:54.987+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:54.988+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:55.003+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:55.009+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:55.009+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:55.010+00:00 [Info] Setting log level to Info 2021-08-16T13:47:55.010+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:55.014+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:55 audit: created new audit service 2021-08-16T13:47:55.020+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:55.020+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:55.020+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:55.020+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:55.020+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:55.032+00:00 [Info] security context initialized 2021-08-16T13:47:55.032+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:55.032+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:55.032+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:55.032+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:55.032+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:55.046+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:55.047+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:55.053+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:55.068+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:55.068+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:55.072+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:55.078+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:55.078+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:55.082+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:55.088+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:55.088+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:55.134+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:55.134+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:47:55.135+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:55.136+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:55.137+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:55.137+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:55.137+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:55.137+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:55.137+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:55.137+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:55.139+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.139+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.144+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:55.144+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:55.144+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:55.144+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.145+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:55.145+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:55.145+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:55.145+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:55.145+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:55.145+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:55.145+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:55.150+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.676905ms 2021-08-16T13:47:55.150+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:55.152+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.152+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:55.163+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:55.163+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:55.163+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:55.163+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:55.163+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400049321f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400049321f pc=0xbd8ea4] goroutine 173 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d829f0 sp=0x4004d829c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d82a30 sp=0x4004d829f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400458d6c0, 0x4004d5a000, 0x11a9890, 0x4004d82b40, 0x400458d6d8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d82ab0 sp=0x4004d82a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400458d6c0, 0x4004d5a000, 0x11a9890, 0x4004d82b40, 0x400458d6d8, 0xffff87241a98, 0x4004d82c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d82af0 sp=0x4004d82ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004466640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d82da0 sp=0x4004d82af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004466640, 0x40002410a0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d82e60 sp=0x4004d82da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d28620, 0x4004d28630, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d83200 sp=0x4004d82e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d28620, 0x4004d28630, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d838f0 sp=0x4004d83200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x400465bed0, 0x4004d1bc00, 0x400025d600, 0x4004d28600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d83fb0 sp=0x4004d838f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d83fb0 sp=0x4004d83fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000203620, 0x4, 0x1136a87, 0xf, 0x400020be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400020c0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400020c108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff8720e1f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c700, 0x4000385000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c700, 0x4000385000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210098, 0x4000385000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000200780, 0x400048a001, 0xdff, 0xdff, 0x40001238e8, 0x6902d8, 0x4000207948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400020c150, 0x400048a001, 0xdff, 0xdff, 0x40002079f8, 0x6f1968, 0x4000207938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400021e840, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400021e840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400021e840, 0xf5a960, 0x40003feec0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40003feea0, 0x400022e600, 0x4000207b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000216870, 0x12f8f98, 0x40003feea0, 0x4000213327, 0x16, 0x4000442900, 0x4000123c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000216870, 0x12f8f98, 0x40003feea0, 0x4000216870, 0x4000212b30, 0x4000212b40, 0x400024cd00, 0x400022e520, 0xee0980, 0x4000442900, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000216870, 0x12f8f98, 0x40003feea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000202420, 0x4000123f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011ff78, 0x4000202420, 0x12db8a8, 0x400020c120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000202420, 0x40002225a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000202420, 0x40002225a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 5 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000249fa0, 0x4, 0x4000249e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000249fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x4000249fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x4000249fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 45 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e7a0, 0x12c18bc, 0x1, 0x40014bfe58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000596c8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40014bffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 23 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 26 [IO wait]: internal/poll.runtime_pollWait(0xffff8720e110, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009a198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009a180, 0x400048b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009a180, 0x400048b000, 0x1000, 0x1000, 0x4000069a88, 0x68f7a4, 0x400021ef70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010050, 0x400048b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x400048b000, 0x1000, 0x1000, 0xf12180, 0x400008a360, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000094960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000094960, 0xa, 0x1008880, 0x400008a320, 0x199, 0xc64, 0x4000069b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000094960, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000096ff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000096ff0, 0x40000a0000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000802c0, 0x40000a0000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000802c0, 0x40000a0000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080300, 0x40000a0000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400021ef20, 0x0, 0x4000069eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400021ef20, 0x0, 0x0, 0x4000069f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400021ef20, 0xee0d40, 0x400008a320, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400023def0, 0x4000094de0, 0x4000038d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 24 [IO wait]: internal/poll.runtime_pollWait(0xffff8720e028, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c080, 0x40001c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c080, 0x40001c0000, 0x1000, 0x1000, 0x40014b2be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010058, 0x40001c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e6c0, 0x40001c0000, 0x1000, 0x1000, 0x40014b2cf8, 0x67867c, 0x400010e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000094ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000094ae0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000258fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 12 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e7a0, 0x1144559, 0x1b, 0x4004c4de38, 0x40046157a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400006d7f0, 0x40046157a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004c4df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044638c0, 0x400006d7f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 11 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40014febe0, 0x4004615680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 8 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000258ec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 10 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000259b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 105 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d630, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c280, 0x40001f1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c280, 0x40001f1000, 0x1000, 0x1000, 0x4000123688, 0x69f178, 0x40001236f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210158, 0x40001f1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027d0e0, 0x40001f1000, 0x1000, 0x1000, 0x5, 0x40001a0000, 0x40001237c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004580840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004580840, 0x68f90a, 0x4000123808, 0x4344e0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004580840, 0x41c330, 0xffff872cb6c0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000aacf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000aacf0, 0x400007f000, 0x1000, 0x1000, 0x4000123998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080640, 0x400007f000, 0x1000, 0x1000, 0x40000945f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080640, 0x400007f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080680, 0x400007f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000123d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000123d80, 0xa, 0x400007f201, 0x1, 0xffff87212880, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000123d80, 0x400007f20a, 0x1, 0x40014c9690, 0x0, 0x400007f259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000123d80, 0x40014c260a, 0x40014c9690, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004410380, 0x400442d360, 0x1f, 0x4000123ee0, 0x4004425d58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4004410380, 0x113075a, 0x7, 0x400442e5f0, 0xd, 0x4004425d58, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4004410340, 0x400442e5f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 70 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 93 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400455b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 32 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40001416e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400044ff80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400009b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff8720dab8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009b118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400009b100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400009b100, 0x0, 0x4000243698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4000131518, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4000131518, 0x12ed9d0, 0x4000131518, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400009b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000215e10, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000215e10, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000215e10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004504a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 79 [select]: net/http.(*persistConn).readLoop(0x400027c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 80 [select]: net/http.(*persistConn).writeLoop(0x400027c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000180380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 99 [select]: net/http.(*persistConn).readLoop(0x400027cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 100 [select]: net/http.(*persistConn).writeLoop(0x400027cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 51 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d718, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c100, 0x4004501000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c100, 0x4004501000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210090, 0x4004501000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027cd80, 0x4004501000, 0x1000, 0x1000, 0x4000102f00, 0x200000003, 0x4000102f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004580300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004580300, 0x40000a600a, 0x40001258c8, 0x4d3b10, 0x4000140060, 0x40000a6000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004580300, 0x0, 0x1, 0x0, 0x4000140078, 0x40000a6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000aa720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000aa720, 0x400048c000, 0x1000, 0x1000, 0x40001259b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080400, 0x400048c000, 0x1000, 0x1000, 0x40003ff5b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080400, 0x400048c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080440, 0x400048c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000125dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000125dd0, 0xa, 0x400048c201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000125dd0, 0x400048c20a, 0x1, 0x400442e5d9, 0x0, 0x400048c2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000125dd0, 0x400049000a, 0x400442e5d9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004410380, 0x400003a3d8, 0x17, 0x4000125f20, 0x40044cec78, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4004410380, 0x113075a, 0x7, 0x40044cec78, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4004410340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d800, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024d298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024d280, 0x40044d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024d280, 0x40044d4000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210598, 0x40044d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027c900, 0x40044d4000, 0x1000, 0x1000, 0x4000103080, 0x200000003, 0x4000103080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001497440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001497440, 0x40000a600a, 0x40014c18c8, 0x4d3b10, 0x4000140060, 0x40000a6000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4001497440, 0x0, 0x1, 0x0, 0x4000140078, 0x40000a6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000aa450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000aa450, 0x40000a1000, 0x1000, 0x1000, 0x40014c19b8, 0x0, 0x40014c1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080340, 0x40000a1000, 0x1000, 0x1000, 0x4000102c00, 0x3, 0x40014c1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080340, 0x40000a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080380, 0x40000a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40014c1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40014c1dd0, 0xa, 0x40000a1201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40014c1dd0, 0x40000a120a, 0x1, 0x400442e37b, 0x0, 0x40000a1263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40014c1dd0, 0x400022e00a, 0x400442e37b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004410380, 0x40000421b0, 0x24, 0x40014c1f20, 0x40044cec60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4004410380, 0x113075a, 0x7, 0x40044cec60, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4004410340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 61 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x400027cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x400027cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 59 [select]: net/http.(*persistConn).readLoop(0x400027d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x400027d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [semacquire]: sync.runtime_Semacquire(0x400465bed8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x400465bed0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400025d600, 0x400011b260, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff4387969, 0x29, 0xfffff43879cb, 0x25, 0x4004615ec0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004be6c30, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff4387969, 0x29, 0xfffff43879cb, 0x25, 0x4004615ec0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004be6c30, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d83190, 0x4004d82e60, 0x400011b260, 0x4000493b00, 0x4000258ec0, 0x0, 0x1bf, 0x4004d82da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000182000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004be6c30, 0xb, 0x4004be6c28, 0x6, 0x4004be6c40, 0xd, 0x4004bbee60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000182000, 0x40003f0000, 0x4000096270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000182000, 0x4000483260, 0x2, 0x2, 0x4000483270, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d9d0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014f8f18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40014f8f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40014f8f00, 0x12d9d48, 0x40014d8640, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bc0e10, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bc0e10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40014d9580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 95 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40014d99c0, 0x40014dad20, 0x40014dad90, 0x4004bcc890, 0x12e8f90, 0x40044647e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400473bf44, 0x5, 0x40014d9580, 0x12eca70, 0x40014d9440, 0x130d478, 0x40002200e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4000188e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4000188e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 96 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40044647e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 97 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40044647e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40000d8090, 0x4004728f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 115 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000080140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 116 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40014febe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 117 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40014febe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40014e83d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40014fec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 120 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40014fec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 121 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40014d99c0, 0x4004bcc890) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 13 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 14 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400014aa00, 0xd00007, 0x400465ac50, 0x5, 0x400006d800, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d460, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009b418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400009b400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400009b400, 0xffff872c0108, 0x30, 0x4004d1c2d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046511b8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046511b8, 0x4004d1c2d0, 0x11ae500, 0x4004464940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044648c0, 0x12ed9d0, 0x40046511b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044648c0, 0x12ed9d0, 0x40046511b8, 0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d378, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009b518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400009b500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400009b500, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046511d0, 0xffff872c0108, 0xffff472bd7e8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046511d0, 0x665e1c, 0x0, 0xfe5e60, 0x4004664dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40046511e8, 0x4004d1c360, 0x11ae500, 0x4004464a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044649a0, 0x12ec710, 0x40046511e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044649a0, 0x12ec710, 0x40046511e8, 0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d8e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024d418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024d400, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024d400, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002105c0, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027cb40, 0x40044f2000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40014978c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40014978c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40014978c0, 0x7356e4, 0xfffff438794e, 0xe, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40044f1da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40044f1da0, 0x40001c5800, 0x200, 0x200, 0xfffff438794e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000259f00, 0x40001c5800, 0x200, 0x200, 0x40001c5800, 0x200, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000259f00, 0x40001c5800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000259f40, 0x40001c5800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045a6f20, 0x0, 0x40014a36b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045a6f20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045a6f20, 0xee0d40, 0x400008ac30, 0x12db068, 0x4000259f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40045a8480, 0x4004581020, 0x40045ba240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff475f8168, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014f9298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40014f9280, 0x4004bff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40014f9280, 0x4004bff000, 0x1000, 0x1000, 0x4004d14be0, 0x469b10, 0x400149a480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014e8430, 0x4004bff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000443680, 0x4004bff000, 0x1000, 0x1000, 0x4004d14cf8, 0x67867c, 0x40047294a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400473cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400473cd80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000443680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 124 [select]: net/http.(*persistConn).writeLoop(0x4000443680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff475f8080, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b7118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b7100, 0x4004d91000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b7100, 0x4004d91000, 0x1000, 0x1000, 0x40014b5be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102b8, 0x4004d91000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004437a0, 0x4004d91000, 0x1000, 0x1000, 0x40014b5cf8, 0x67867c, 0x4004dc6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004b7f0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004b7f0e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40004437a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 131 [select]: net/http.(*persistConn).writeLoop(0x40004437a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 148 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000161980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 149 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045de000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 150 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045de000, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40045de000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 152 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 153 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e7a0, 0x1144559, 0x1b, 0x4004597e38, 0x40045baa20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400020fe70, 0x40045baa20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004597f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004581500, 0x400020fe70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 158 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d548, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40045f2098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40045f2080, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40045f2080, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210670, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027de60, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004581d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004581d40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000054438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004581d40, 0x0, 0x4000054478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045c4e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045c4e40, 0x40045bea00, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161b80, 0x40045bea00, 0x200, 0x200, 0x40044d1987, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161b80, 0x40045bea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161bc0, 0x40045bea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045dcc60, 0x0, 0x40000546b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045dcc60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045dcc60, 0xee0d40, 0x400008b090, 0x12db068, 0x4000161bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40045a9050, 0x40045fa120, 0x40045bb800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 156 [select]: net/http.(*persistConn).readLoop(0x400027de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 157 [select]: net/http.(*persistConn).writeLoop(0x400027de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001619c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d34200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004d34200, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 170 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 171 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 172 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:00.323+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:00.323+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:00.328+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:00.332+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:00.332+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:00.332+00:00 [Info] Setting log level to Info 2021-08-16T13:48:00.332+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:00.335+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:48:00.343+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:00.343+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:00.343+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:00.343+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:00.343+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021/08/16 13:48:00 audit: created new audit service 2021-08-16T13:48:00.355+00:00 [Info] security context initialized 2021-08-16T13:48:00.355+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:00.355+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:00.355+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:00.355+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:00.355+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:00.368+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:00.368+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:00.381+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:00.394+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:00.394+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:00.396+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:00.397+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:00.401+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:00.406+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:00.408+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:00.408+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:00.448+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:00.449+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 55 us 2021-08-16T13:48:00.449+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:00.451+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:00.451+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:00.452+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:00.452+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:00.452+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:00.452+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:00.452+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:00.458+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.458+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.459+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:00.459+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:00.459+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:00.459+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.459+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:00.459+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:00.459+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:00.459+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:00.459+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:00.459+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:00.459+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:00.464+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.899639ms 2021-08-16T13:48:00.465+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:00.466+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:00.466+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:00.477+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:00.477+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:00.477+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:00.477+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:00.478+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004cd0fdf fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004cd0fdf pc=0xbd8ea4] goroutine 170 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004c729f0 sp=0x4004c729c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004c72a30 sp=0x4004c729f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4000405a00, 0x40045fd200, 0x11a9890, 0x4004c72b40, 0x4000405a18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004c72ab0 sp=0x4004c72a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4000405a00, 0x40045fd200, 0x11a9890, 0x4004c72b40, 0x4000405a18, 0xc18f8c, 0x4004c72c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004c72af0 sp=0x4004c72ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000224b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004c72da0 sp=0x4004c72af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000224b40, 0x400440d6c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004c72e60 sp=0x4004c72da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cb5b40, 0x4004cb5b50, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004c73200 sp=0x4004c72e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cb5b40, 0x4004cb5b50, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004c738f0 sp=0x4004c73200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dabe30, 0x4004cff880, 0x4000250680, 0x4004cb5b20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004c73fb0 sp=0x4004c738f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004c73fb0 sp=0x4004c73fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40002054a0, 0x4, 0x1136a87, 0xf, 0x400019de68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400020e108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400020e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011e080, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011e080, 0x4000139000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110008, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001120c0, 0x4000138001, 0xdff, 0xdff, 0x40000918e8, 0x6902d8, 0x4000153948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000118030, 0x4000138001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000136160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000136160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000136160, 0xf5a960, 0x4000082860, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082840, 0x4000046240, 0x4000153b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40000ac000, 0x12f8f98, 0x4000082840, 0x4000036b67, 0x16, 0x400000c5a0, 0x4000091c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40000ac000, 0x12f8f98, 0x4000082840, 0x40000ac000, 0x40000361a0, 0x40000361d0, 0x400011e000, 0x4000046020, 0xee0980, 0x400000c5a0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40000ac000, 0x12f8f98, 0x4000082840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000204420, 0x4000091f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019bf78, 0x4000204420, 0x12db8a8, 0x4000118000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000204420, 0x40002223c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000204420, 0x40002223c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 7 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001ab340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000212198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000212180, 0x40001fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000212180, 0x40001fa000, 0x1000, 0x1000, 0x4000019a88, 0x68f7a4, 0x400021e730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a2048, 0x40001fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400026c480, 0x40001fa000, 0x1000, 0x1000, 0xf12180, 0x4000180130, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000192540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000192540, 0xa, 0x1008880, 0x40001800f0, 0x199, 0x1000, 0x4000019b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000192540, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000194b70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000194b70, 0x40000e8000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001aa100, 0x40000e8000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001aa100, 0x40000e8000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001aa140, 0x40000e8000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400021e6e0, 0x0, 0x4000019eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400021e6e0, 0x0, 0x0, 0x4000019f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400021e6e0, 0xee0d40, 0x40001800f0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001261b0, 0x40001926c0, 0x4000038420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 54 [select]: net/http.(*persistConn).readLoop(0x400026c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x400026c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 9 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002066e0, 0x12c18bc, 0x1, 0x400019de58, 0x4000182720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000f6ec8, 0x4000182720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001aa0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 10 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001aa0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 109 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a960f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011f000, 0x400251f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011f000, 0x400251f000, 0x1000, 0x1000, 0x400256ea48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000106e8, 0x400251f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d320, 0x400251f000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x40000ac6e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400252a2a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400252a2a0, 0x400023190a, 0x2, 0x4, 0x4000231920, 0x2, 0x4002370438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400252a2a0, 0x7356e4, 0xffffd9e6f94e, 0xe, 0x4, 0x40002319c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40046cb1d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40046cb1d0, 0x4004c33200, 0x200, 0x200, 0xffffd9e6f94e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400247f580, 0x4004c33200, 0x200, 0x200, 0x4000043537, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400247f580, 0x4004c33200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400247f5c0, 0x4004c33200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c358c0, 0x4000231d00, 0x40023706b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c358c0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c358c0, 0xee0d40, 0x40000acff0, 0x12db068, 0x400247f5c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046cd5f0, 0x400252ba40, 0x4004c04b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 13 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000c8098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000c8080, 0x40000ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000c8080, 0x40000ca000, 0x1000, 0x1000, 0x4000013be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110060, 0x40000ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400001c360, 0x40000ca000, 0x1000, 0x1000, 0x4000013cf8, 0x67867c, 0x4000182c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001126c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001126c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400001c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 14 [select]: net/http.(*persistConn).writeLoop(0x400001c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001aaf40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 108 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400247e400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000212d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000212d00, 0x400445e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000212d00, 0x400445e000, 0x1000, 0x1000, 0x4000467688, 0x69f178, 0x40004676f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a2098, 0x400445e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400441c360, 0x400445e000, 0x1000, 0x1000, 0xffff885bafff, 0x4002402500, 0x40004677e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fbce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fbce0, 0xa, 0xffff883e3c00, 0xffff885bafff, 0x4000467838, 0x41cff4, 0xffff883e3b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fbce0, 0x41c330, 0xffffb1aa2ea0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024a8e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024a8e40, 0x4004468000, 0x1000, 0x1000, 0x4000467998, 0x44e884, 0x4000467948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001aa800, 0x4004468000, 0x1000, 0x1000, 0x4000082e98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001aa800, 0x4004468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001aa880, 0x4004468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000467d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000467d80, 0xa, 0x4004468201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000467d80, 0x400446820a, 0x1, 0x40023f3d9a, 0x0, 0x4004468259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000467d80, 0x400021f00a, 0x40023f3d9a, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000dae10, 0x40044f17c0, 0x1f, 0x4000467ee0, 0x40044f3320, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000dae10, 0x113075a, 0x7, 0x4002565b90, 0xd, 0x40044f3320, 0x0, 0x4714f4, 0x40000dae10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000dadd0, 0x4002565b90, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 31 [select]: net/http.(*persistConn).readLoop(0x400441c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 32 [select]: net/http.(*persistConn).writeLoop(0x400441c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40046c91e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 33 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002066e0, 0x1144559, 0x1b, 0x4000175e38, 0x4004610420, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40024a7a50, 0x4004610420, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x40024a8b70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000175f80, 0x0, 0x1000000010000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004516900, 0x40024a7a50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 103 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001aa580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000112ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400043e9f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000c8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000c8718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000c8700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000c8700, 0x0, 0x4000059698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044a3a70, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044a3a70, 0x12ed9d0, 0x40044a3a70, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000c8180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d450, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d450, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400013d450) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400043fe60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 87 [select]: net/http.(*persistConn).writeLoop(0x40004590e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450180, 0x40000e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450180, 0x40000e9000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a2088, 0x40000e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000458900, 0x40000e9000, 0x1000, 0x1000, 0x4000241e00, 0x200000003, 0x4000241e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fbc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fbc20, 0x400040000a, 0x40000918c8, 0x4d3b10, 0x40000820c0, 0x4000400000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fbc20, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000400000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000478a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000478a80, 0x400257c000, 0x1000, 0x1000, 0x40000919b8, 0x0, 0x4000091988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000210500, 0x400257c000, 0x1000, 0x1000, 0x4000186f00, 0x3, 0x4000091be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000210500, 0x400257c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000210580, 0x400257c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000091dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000091dd0, 0xa, 0x400257c201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000091dd0, 0x400257c20a, 0x1, 0x40023f388b, 0x0, 0x400257c263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000091dd0, 0x400020020a, 0x40023f388b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000dae10, 0x4000042ae0, 0x24, 0x4000091f20, 0x400255ca20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000dae10, 0x113075a, 0x7, 0x400255ca20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000dadd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000276690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a963a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450280, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450280, 0x40024fe000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000208158, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000458d80, 0x40024fe000, 0x1000, 0x1000, 0x4000102f00, 0x200000003, 0x4000102f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000192d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000192d20, 0x400040000a, 0x40001978c8, 0x4d3b10, 0x40000820c0, 0x4000400000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000192d20, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000400000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000478cf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000478cf0, 0x4002377000, 0x1000, 0x1000, 0x40001979b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000210640, 0x4002377000, 0x1000, 0x1000, 0x4000082e98, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000210640, 0x4002377000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000210680, 0x4002377000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000197dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000197dd0, 0xa, 0x4002377201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000197dd0, 0x400237720a, 0x1, 0x4002565b79, 0x0, 0x40023772e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000197dd0, 0x40000db60a, 0x4002565b79, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000dae10, 0x40044084b0, 0x17, 0x4000197f20, 0x400019e660, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000dae10, 0x113075a, 0x7, 0x400019e660, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000dadd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 29 [select]: net/http.(*persistConn).readLoop(0x4000458900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [select]: net/http.(*persistConn).writeLoop(0x4000458900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 75 [select]: net/http.(*persistConn).readLoop(0x4000458d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 76 [select]: net/http.(*persistConn).writeLoop(0x4000458d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [select]: net/http.(*persistConn).readLoop(0x40004590e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff8833b968, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450980, 0x4004c76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450980, 0x4004c76000, 0x1000, 0x1000, 0x400256fbe0, 0x469b10, 0x4000187800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010878, 0x4004c76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400026c900, 0x4004c76000, 0x1000, 0x1000, 0x400256fcf8, 0x67867c, 0x4004c050e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c442a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c442a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400026c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [select]: net/http.(*persistConn).readLoop(0x400000d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x400000d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 79 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400247eb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a961d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004450680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004450680, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c8b590, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c8b590, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400440b380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 81 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400440b7c0, 0x400440cc40, 0x400440ccb0, 0x4004cb46e0, 0x12e8f90, 0x40044c28c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045ef804, 0x5, 0x400440b380, 0x12eca70, 0x400440b240, 0x130d478, 0x40000d8310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400043c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400043c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40044c28c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 115 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40044c28c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002456750, 0x4004587320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40001aa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004428a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004428a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000208498) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 121 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004428aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 122 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004428aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 123 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400440b7c0, 0x4004cb46e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 124 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004428a00, 0x4004587440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40001805f0, 0xd00007, 0x40024725d0, 0x5, 0x40024a7a60, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000212f98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000212f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000212f80, 0xffffb1b12a68, 0x30, 0x4004cf01b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004616210, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004616210, 0x4004cf01b0, 0x11ae500, 0x40000e0860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40000e07e0, 0x12ed9d0, 0x4004616210, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40000e07e0, 0x12ed9d0, 0x4004616210, 0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000213118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000213100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000213100, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004616228, 0xffffb1b12a68, 0xffff88331e08, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004616228, 0x665e1c, 0x0, 0xfe5e60, 0x40000f6dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004616240, 0x4004cf0240, 0x11ae500, 0x40000e0940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40000e08c0, 0x12ec710, 0x4004616240, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40000e08c0, 0x12ec710, 0x4004616240, 0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 135 [semacquire]: sync.runtime_Semacquire(0x4004dabe38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dabe30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000250680, 0x400043e5a0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd9e6f969, 0x29, 0xffffd9e6f9cb, 0x25, 0x4004db26c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cbe4f0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd9e6f969, 0x29, 0xffffd9e6f9cb, 0x25, 0x4004db26c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cbe4f0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004c73190, 0x4004c72e60, 0x400043e5a0, 0x4000227d40, 0x40001aaf40, 0x0, 0x1bf, 0x4000468da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001c0000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004cbe4f0, 0xb, 0x4004cbe4e8, 0x6, 0x4004cbe500, 0xd, 0x4004c8ce00, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001c0000, 0x40003f0000, 0x4000194360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001c0000, 0x4000116620, 0x2, 0x2, 0x4000116630, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 148 [select]: net/http.(*persistConn).writeLoop(0x400026c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001abac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 156 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40046c91e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 157 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40046c91e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 137 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 138 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 139 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 140 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40001aba80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 89 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002066e0, 0x1144559, 0x1b, 0x4000097e38, 0x4004611020, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40047a26c0, 0x4004611020, 0x4000458d80, 0x40001b1640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12c2e00, 0x4000472e01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000097f80, 0x12c2e00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004517140, 0x40047a26c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff8833b880, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000c9318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000c9300, 0x40047d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000c9300, 0x40047d3000, 0x1000, 0x1000, 0x4002424be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110230, 0x40047d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400001d440, 0x40047d3000, 0x1000, 0x1000, 0x4002424cf8, 0x67867c, 0x4000039bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004732f60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004732f60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400001d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x400001d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a962c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450380, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450380, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110148, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004590e0, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001139e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001139e0, 0x40003e960a, 0x40003f9b00, 0x4000233ce0, 0x400027e420, 0x400027fbc0, 0x40000f8c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001139e0, 0x40003f9800, 0x40000f8c78, 0x655784, 0x40003fa8a0, 0x400027e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40046509f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40046509f0, 0x400460d600, 0x200, 0x200, 0x4, 0x0, 0x40000f8d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001abb40, 0x400460d600, 0x200, 0x200, 0x40001ad9b7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001abb40, 0x400460d600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001abb80, 0x400460d600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004657080, 0x4000103500, 0x40000f8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004657080, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004657080, 0xee0d40, 0x4000180910, 0x12db068, 0x40001abb80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004652750, 0x40045172c0, 0x4004611560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c96f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c96f00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 169 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:05.637+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:05.637+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:05.650+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:05.653+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:05.653+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:05.653+00:00 [Info] Setting log level to Info 2021-08-16T13:48:05.655+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:48:05.656+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021/08/16 13:48:05 audit: created new audit service 2021-08-16T13:48:05.665+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:05.665+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:05.665+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:05.665+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:05.665+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:05.678+00:00 [Info] security context initialized 2021-08-16T13:48:05.678+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:05.678+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:05.678+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:05.678+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:05.678+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:05.691+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:05.693+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:05.693+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:05.719+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:05.720+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:05.721+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:05.721+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:05.723+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:05.724+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:05.725+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:05.725+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:05.772+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:05.772+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 51 us 2021-08-16T13:48:05.772+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:05.774+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:05.774+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:05.775+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:05.775+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:05.775+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:05.775+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:05.775+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:05.777+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.777+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.780+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:05.780+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:05.780+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:05.780+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.780+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:05.780+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:05.780+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:05.780+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:05.780+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:05.780+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:05.780+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:05.785+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.53796ms 2021-08-16T13:48:05.785+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:05.786+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.786+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004cc0d9f fatal error: fault 2021-08-16T13:48:05.797+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:05.797+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:05.798+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:05.798+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:05.798+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004cc0d9f pc=0xbd8ea4] goroutine 204 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d789f0 sp=0x4004d789c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d78a30 sp=0x4004d789f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004ce2680, 0x400481b800, 0x11a9890, 0x4004d78b40, 0x4004ce2698, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d78ab0 sp=0x4004d78a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004ce2680, 0x400481b800, 0x11a9890, 0x4004d78b40, 0x4004ce2698, 0xc18f8c, 0x4004d78c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d78af0 sp=0x4004d78ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40001af040, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d78da0 sp=0x4004d78af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40001af040, 0x400007fa40, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d78e60 sp=0x4004d78da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cd2e50, 0x4004cd2e60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d79200 sp=0x4004d78e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cd2e50, 0x4004cd2e60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d798f0 sp=0x4004d79200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004cff700, 0x4004eb6a80, 0x40000c3b80, 0x4004cd2e30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d79fb0 sp=0x4004d798f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d79fb0 sp=0x4004d79fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000958c0, 0x4, 0x1136a87, 0xf, 0x4000119e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012a098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012a080, 0x4000145000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012a080, 0x4000145000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e010, 0x4000145000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e120, 0x4000216001, 0xdff, 0xdff, 0x400019d8e8, 0x6902d8, 0x4000115948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400011a048, 0x4000216001, 0xdff, 0xdff, 0x40001159f8, 0x6f1968, 0x4000115938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e6840, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6840, 0xf5a960, 0x40001ba440, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001ba420, 0x40001c22c0, 0x4000115b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40001801e0, 0x12f8f98, 0x40001ba420, 0x40001b4c37, 0x16, 0x40001c7200, 0x400019dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40001801e0, 0x12f8f98, 0x40001ba420, 0x40001801e0, 0x40001b4448, 0x40001b4450, 0x4000208080, 0x40001c21e0, 0xee0980, 0x40001c7200, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40001801e0, 0x12f8f98, 0x40001ba420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000094840, 0x400019df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019df78, 0x4000094840, 0x12db8a8, 0x400011a018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000094840, 0x40000a4460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000094840, 0x40000a4460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40004357a0, 0x4, 0x4000435668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40004357a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x40004357a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x40004357a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7d0, 0x12c18bc, 0x1, 0x4000243e58, 0x4000130720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000433ec8, 0x4000130720, 0x4000433ec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000243fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001220c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001220c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096180, 0x4000144000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096180, 0x4000144000, 0x1000, 0x1000, 0x4000232a88, 0x68f7a4, 0x40001421b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010058, 0x4000144000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000144000, 0x1000, 0x1000, 0xf12180, 0x40001241d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003eb140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003eb140, 0xa, 0x1008880, 0x4000124190, 0x199, 0x1000, 0x4000232b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003eb140, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000094300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000094300, 0x400046e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001434000, 0x400046e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001434000, 0x400046e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001434040, 0x400046e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000142160, 0x0, 0x4000232eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000142160, 0x0, 0x0, 0x4000232f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000142160, 0xee0d40, 0x4000124190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4001428000, 0x4000082480, 0x4000130780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 32 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000081700, 0x4004626340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096280, 0x400017a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096280, 0x400017a000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100a0, 0x400017a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000ca20, 0x400017a000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x40000389c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003eb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003eb560, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000096e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012a798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012a780, 0x4002400000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012a780, 0x4002400000, 0x1000, 0x1000, 0x4000425be0, 0x469b10, 0x4001417500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e088, 0x4002400000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6480, 0x4002400000, 0x1000, 0x1000, 0x4000425cf8, 0x67867c, 0x40034609c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4003444780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4003444780, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 24 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000122e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001d9a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400249d230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 48 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400010eea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 57 [IO wait]: internal/poll.runtime_pollWait(0xffffaec122c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001466198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001466180, 0x4001468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001466180, 0x4001468000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001440050, 0x4001468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x4001468000, 0x1000, 0x1000, 0x4001416c00, 0x200000003, 0x4001416c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ebce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003ebce0, 0x400143c00a, 0x40014618c8, 0x4d3b10, 0x40001ba060, 0x400143c000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003ebce0, 0x0, 0x1, 0x0, 0x40001ba078, 0x400143c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4003446660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4003446660, 0x40024fd000, 0x1000, 0x1000, 0x40014619b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080400, 0x40024fd000, 0x1000, 0x1000, 0x4000082898, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080400, 0x40024fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080440, 0x40024fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4001461dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001461dd0, 0xa, 0x40024fd201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4001461dd0, 0x40024fd20a, 0x1, 0x4001414829, 0x0, 0x40024fd2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4001461dd0, 0x40024a510a, 0x4001414829, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001472f0, 0x4001432120, 0x17, 0x4001461f20, 0x4001444210, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40001472f0, 0x113075a, 0x7, 0x4001444210, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40001472b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 70 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 95 [IO wait]: internal/poll.runtime_pollWait(0xffffaec121d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000097418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000097400, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000097400, 0x40023f9000, 0x1000, 0x1000, 0x400019f688, 0x69f178, 0x400019f6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8680, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027a900, 0x40023f9000, 0x1000, 0x1000, 0x5, 0x400015ef00, 0x400019f7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024822a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024822a0, 0x68f90a, 0x4000245080, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024822a0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400041b1a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400041b1a0, 0x400248b000, 0x1000, 0x1000, 0x400019f998, 0x44e884, 0x400019f948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000122fc0, 0x400248b000, 0x1000, 0x1000, 0x4000082898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000122fc0, 0x400248b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000123000, 0x400248b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019fd80, 0xa, 0x400248b201, 0x1, 0xffffaec10648, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019fd80, 0x400248b20a, 0x1, 0x40001b5cb0, 0x0, 0x400248b259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019fd80, 0x400246210a, 0x40001b5cb0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001472f0, 0x4000470f00, 0x1f, 0x400019fee0, 0x400046dad0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40001472f0, 0x113075a, 0x7, 0x4001414840, 0xd, 0x400046dad0, 0x0, 0x2a, 0x4000053fa8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40001472b0, 0x4001414840, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 31 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400047aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 80 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001434a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x40001c7c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x40001c7c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096f98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000096f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000096f80, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400046d710, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400046d710, 0x12ed9d0, 0x400046d710, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000096f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40024a5040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40024a5040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40024a5040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400249def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 30 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400047aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 99 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000122b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 90 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000f4700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffffaec123a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001466098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001466080, 0x400146a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001466080, 0x400146a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001440058, 0x400146a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x400146a000, 0x1000, 0x1000, 0x4000102d80, 0x200000003, 0x4000102d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ebe00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003ebe00, 0x400143c00a, 0x40014658c8, 0x4d3b10, 0x40001ba060, 0x400143c000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003ebe00, 0x0, 0x1, 0x0, 0x40001ba078, 0x400143c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000194d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000194d50, 0x4001437000, 0x1000, 0x1000, 0x40014659b8, 0x0, 0x4001465988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000122840, 0x4001437000, 0x1000, 0x1000, 0x4000102a80, 0x3, 0x4001465be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000122840, 0x4001437000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000122a00, 0x4001437000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4001465dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001465dd0, 0xa, 0x4001437201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4001465dd0, 0x400143720a, 0x1, 0x40024729bb, 0x0, 0x4001437263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4001465dd0, 0x400004620a, 0x40024729bb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001472f0, 0x4001430210, 0x24, 0x4001465f20, 0x40014441e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40001472f0, 0x113075a, 0x7, 0x40014441e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40001472b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014343c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 65 [select]: net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 64 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 29 [select]: net/http.(*persistConn).writeLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 114 [select]: net/http.(*persistConn).readLoop(0x400027a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x400027a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 81 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209180, 0x58ec24, 0x4000183020, 0x4000432f78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c88a50, 0x4714f4, 0x4000183020, 0x4000432f78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c88a50, 0x4001414500, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004449500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 130 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000081700, 0x400007f340, 0x400007f3b0, 0x4004626340, 0x12e8f90, 0x40046b02a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004843be4, 0x5, 0x4004449500, 0x12eca70, 0x4004449380, 0x130d478, 0x40001f8150, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400348ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400348ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40046b02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 132 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40046b02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 133 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024a7320, 0x400452df80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000122b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004caa0a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 136 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004caa0a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010598) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 138 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004caa0a0, 0x4004cc6000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 139 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7d0, 0x1144559, 0x1b, 0x4002465e38, 0x4004cc6120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cd2180, 0x4004cc6120, 0xf1f700, 0x4000468050) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002465f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004527b00, 0x4004cd2180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 140 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 141 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000180ff0, 0xd00007, 0x4004caf648, 0x5, 0x4004cd2190, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffffaec120f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209400, 0xffffaec8e108, 0x30, 0x4004ca9470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c88ff0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c88ff0, 0x4004ca9470, 0x11ae500, 0x400348ada0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400348ad20, 0x12ed9d0, 0x4004c88ff0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400348ad20, 0x12ed9d0, 0x4004c88ff0, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 143 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209500, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c89008, 0xffffaec8e108, 0xffff840abcc8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c89008, 0x665e1c, 0x0, 0xfe5e60, 0x4004cdbdc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c89020, 0x4004ca9500, 0x11ae500, 0x400348ae80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400348ae00, 0x12ec710, 0x4004c89020, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400348ae00, 0x12ec710, 0x4004c89020, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012a418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012a400, 0x4001442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012a400, 0x4001442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001440008, 0x4001442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7c20, 0x4001442000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082660, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40024e5c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082660, 0x0, 0x40024e5c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400478b110) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400478b110, 0x4004793800, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014350c0, 0x4004793800, 0x200, 0x200, 0x4001438187, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014350c0, 0x4004793800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001435100, 0x4004793800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047ca9a0, 0x0, 0x40024e5eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047ca9a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047ca9a0, 0xee0d40, 0x4000124820, 0x12db068, 0x4001435100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400479abd0, 0x400147b1a0, 0x400478d3e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff845d11a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012b598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012b580, 0x4004dad000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012b580, 0x4004dad000, 0x1000, 0x1000, 0x4004606ad8, 0x412bc0, 0x1c161d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e408, 0x4004dad000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d9e480, 0x4004dad000, 0x1000, 0x1000, 0x4004cdc000, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004daf080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004daf080, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004cdbc18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004daf080, 0x412bc0, 0x0, 0x0, 0x4004cdbcc8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ca98c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ca98c0, 0x4004ce0200, 0x200, 0x200, 0x4004ca94d0, 0x30, 0x4004cdbd78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004449a80, 0x4004ce0200, 0x200, 0x200, 0x0, 0xffffaec8e108, 0x4004cdbd68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004449a80, 0x4004ce0200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004449ac0, 0x4004ce0200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004ce8420, 0x12ec700, 0x4004cdbeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004ce8420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004ce8420, 0xee0d40, 0x4000181180, 0x12db068, 0x4004449ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004cbe7e0, 0x4004cdc180, 0x4004cc6660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 161 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 151 [IO wait]: internal/poll.runtime_pollWait(0xffff845d1378, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012b218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012b200, 0x4004d83000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012b200, 0x4004d83000, 0x1000, 0x1000, 0x4004603be0, 0x469b10, 0x4000186f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e310, 0x4004d83000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7b00, 0x4004d83000, 0x1000, 0x1000, 0x4004603cf8, 0x67867c, 0x400463cc00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4003445b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4003445b00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 152 [select]: net/http.(*persistConn).writeLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 160 [semacquire]: sync.runtime_Semacquire(0x4004cff708) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004cff700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000c3b80, 0x400041ac60, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffde12f969, 0x29, 0xffffde12f9cb, 0x25, 0x4004cc6d20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cae980, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffde12f969, 0x29, 0xffffde12f9cb, 0x25, 0x4004cc6d20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cae980, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d79190, 0x4004d78e60, 0x400041ac60, 0x40003cd440, 0x4000122e80, 0x0, 0x13f, 0x400023cda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000154000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004cae980, 0xb, 0x4004cae978, 0x6, 0x4004cae990, 0xd, 0x4004c874e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000154000, 0x40003e0000, 0x4000110210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000154000, 0x400022e990, 0x2, 0x2, 0x400022e9a0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 165 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000081e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 177 [IO wait]: internal/poll.runtime_pollWait(0xffff845d1290, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012b498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012b480, 0x4004dc9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012b480, 0x4004dc9000, 0x1000, 0x1000, 0x4004db5be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e410, 0x4004dc9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d9e360, 0x4004dc9000, 0x1000, 0x1000, 0x4004db5cf8, 0x67867c, 0x4004dc62a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004daf1a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004daf1a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004d9e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400467c6e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400467c6e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x400467c6e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 172 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7d0, 0x1144559, 0x1b, 0x400246be38, 0x400463d560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004626d10, 0x400463d560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400246bf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004dae360, 0x4004626d10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 178 [select]: net/http.(*persistConn).writeLoop(0x4004d9e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 175 [select]: net/http.(*persistConn).readLoop(0x4004d9e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 176 [select]: net/http.(*persistConn).writeLoop(0x4004d9e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 145 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 194 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 195 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 196 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 197 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 198 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004577c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 199 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004577c00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 200 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 201 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 202 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 203 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:10.946+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:10.946+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:10.956+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:10.957+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:10.957+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:10.957+00:00 [Info] Setting log level to Info 2021-08-16T13:48:10.957+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:10.960+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:10 audit: created new audit service 2021-08-16T13:48:10.968+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:10.968+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:10.969+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:10.969+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:10.969+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:10.979+00:00 [Info] security context initialized 2021-08-16T13:48:10.979+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:10.979+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:10.979+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:10.979+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:10.979+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:10.991+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:10.992+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:10.995+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:11.013+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:11.013+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:11.015+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:11.017+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:11.035+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:11.044+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:11.044+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:11.045+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:11.083+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:11.083+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 62 us 2021-08-16T13:48:11.083+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:11.085+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:11.085+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:11.104+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:11.105+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:11.105+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:11.106+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:11.106+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:11.106+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.107+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.108+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:11.108+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:11.108+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:11.108+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.109+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:11.109+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:11.109+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:11.109+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:11.109+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:11.109+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:11.109+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:11.113+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.059546ms 2021-08-16T13:48:11.113+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:11.114+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.114+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:11.125+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:11.126+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:11.126+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:11.126+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:11.126+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:48:11.126+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.126+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400245b45f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400245b45f pc=0xbd8ea4] goroutine 111 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400042c9f0 sp=0x400042c9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400042ca30 sp=0x400042c9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400443d040, 0x4004c09800, 0x11a9890, 0x400042cb40, 0x400443d058, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400042cab0 sp=0x400042ca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400443d040, 0x4004c09800, 0x11a9890, 0x400042cb40, 0x400443d058, 0xc18f8c, 0x400042cc18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400042caf0 sp=0x400042cab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032c80, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x400042cda0 sp=0x400042caf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032c80, 0x4004430f50, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x400042ce60 sp=0x400042cda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004bf05f0, 0x4004bf0600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x400042d200 sp=0x400042ce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004bf05f0, 0x4004bf0600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x400042d8f0 sp=0x400042d200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004be3840, 0x4004bbfc00, 0x40003fc000, 0x4004bf05d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x400042dfb0 sp=0x400042d8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400042dfb0 sp=0x400042dfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fbdd0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff886c5198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x40001a8001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x40001a8001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000cec60, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000cec60, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000cec60, 0xf5a960, 0x4000082da0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082d80, 0x40000467c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c820, 0x12f8f98, 0x4000082d80, 0x4000037447, 0x16, 0x400000d680, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400008c820, 0x4000036c58, 0x4000036c60, 0x40000f0900, 0x40000466e0, 0xee0980, 0x400000d680, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c820, 0x12f8f98, 0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1400, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 11 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 12 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 13 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 14 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 15 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000410000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 34 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000537a0, 0x4, 0x4000053668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000537a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000537a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000537a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x12c18bc, 0x1, 0x400039fe58, 0x40000383c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004acec8, 0x40000383c0, 0x40004acec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400039ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001f8400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x40001ba360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x40001ba360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001f8400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 45 [IO wait]: internal/poll.runtime_pollWait(0xffff886c50b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40001a9000, 0x1000, 0x1000, 0x40004c4a88, 0x68f7a4, 0x40000ce730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100e0, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ba360, 0x40001a9000, 0x1000, 0x1000, 0xf12180, 0x400008c400, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033f620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400033f620, 0xa, 0x1008880, 0x400008c3c0, 0x199, 0x1000, 0x40004c4b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400033f620, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400036d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400036d560, 0x40004c6000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f8640, 0x40004c6000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f8640, 0x40004c6000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f8680, 0x40004c6000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000ce6e0, 0x0, 0x40004c4eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce6e0, 0x0, 0x0, 0x40004c4f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce6e0, 0xee0d40, 0x400008c3c0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000de6c0, 0x400033fce0, 0x4000038c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x40004bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x40004bc000, 0x1000, 0x1000, 0x400006bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010410, 0x40004bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ba7e0, 0x40004bc000, 0x1000, 0x1000, 0x400006bcf8, 0x67867c, 0x4000038ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033fb60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400033fb60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x40001ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 23 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004afc118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004afc100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004afc100, 0x58ec48, 0x400250c960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cd3500, 0x4714f4, 0x400250c960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cd3500, 0x400250ed00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004c8c4c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 97 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f9f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 53 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 58 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012c980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 49 [select]: net/http.(*persistConn).readLoop(0x40001bb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 87 [IO wait]: internal/poll.runtime_pollWait(0xffff886c47a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004420298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004420280, 0x400459b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004420280, 0x400459b000, 0x1000, 0x1000, 0x4002543688, 0x69f178, 0x40025436f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400440a080, 0x400459b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400441c900, 0x400459b000, 0x1000, 0x1000, 0x5, 0x4002402500, 0x40025437c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044249c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044249c0, 0x68f90a, 0x4004418c60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40044249c0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023f5230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023f5230, 0x40045f0000, 0x1000, 0x1000, 0x4002543998, 0x44e884, 0x4002543948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004414900, 0x40045f0000, 0x1000, 0x1000, 0x40004ff738, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004414900, 0x40045f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004414940, 0x40045f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002543d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002543d80, 0xa, 0x40045f0201, 0x1, 0xffff884ba628, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002543d80, 0x40045f020a, 0x1, 0x4004b01460, 0x0, 0x40045f0259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002543d80, 0x40045c960a, 0x4004b01460, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002500ee0, 0x4004562000, 0x1f, 0x4002543ee0, 0x4004560000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4002500ee0, 0x113075a, 0x7, 0x40046665f0, 0xd, 0x4004560000, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4002500ea0, 0x40046665f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 59 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000401b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40024aa900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 61 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40001ac900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4888, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001aca18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001aca00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001aca00, 0x0, 0x40023dfe98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024a5b60, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024a5b60, 0x12ed9d0, 0x40024a5b60, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40001ac980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099c70, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099c70, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000099c70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40024ab3e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f69a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 186 [IO wait]: internal/poll.runtime_pollWait(0xffff88698f98, 0x77, 0x4000103b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000453c98, 0x77, 0x12f8600, 0x4004f60720, 0x4000453c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4000453c80, 0x12f8668, 0x4004f60720, 0x0, 0x0, 0x12db428, 0x4004f5ba60, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4000453c80, 0x12f8668, 0x4004f60720, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c4b0, 0x0, 0xffffb043f6d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004f60720, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004f60720, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c4b0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4000453c00, 0x12f8668, 0x4004f60720, 0x0, 0x4004f6c4b0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4000453c00, 0x12f8668, 0x4004f60720, 0x0, 0x4004f6c4b0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4000453c00, 0x12f8668, 0x4004f60720, 0x12e9300, 0x4004f6c4b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4000453c00, 0x12f8668, 0x4004f60720, 0x4004ed9280, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40000822a0, 0x12f8668, 0x4004f60720, 0x112eab2, 0x3, 0x4004ec3eb0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004f604e0, 0x112eab2, 0x3, 0x4004ec3eb0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004f604e0, 0x0, 0x4004f12930, 0x4, 0x4004ec3eb0, 0xe, 0x0, 0x4004eccc60, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004f5e420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x40001bb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [select]: net/http.(*persistConn).readLoop(0x400441c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: net/http.(*persistConn).writeLoop(0x400441c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 25 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f8d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffff886c45d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000452a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000452a80, 0x400032f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000452a80, 0x400032f000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000428190, 0x400032f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001bbb00, 0x400032f000, 0x1000, 0x1000, 0x4000412900, 0x200000003, 0x4000412900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011b560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011b560, 0x400043800a, 0x40003a18c8, 0x4d3b10, 0x40000820c0, 0x4000438000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400011b560, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000438000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400458b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400458b8c0, 0x40004c7000, 0x1000, 0x1000, 0x40003a19b8, 0x0, 0x40003a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f9680, 0x40004c7000, 0x1000, 0x1000, 0x4000102900, 0x3, 0x40003a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f9680, 0x40004c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f96c0, 0x40004c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40003a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a1dd0, 0xa, 0x40004c7201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40003a1dd0, 0x40004c720a, 0x1, 0x40046665d9, 0x0, 0x40004c72e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40003a1dd0, 0x40044df70a, 0x40046665d9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002500ee0, 0x4000420dc8, 0x17, 0x40003a1f20, 0x400011ce10, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4002500ee0, 0x113075a, 0x7, 0x400011ce10, 0x0, 0x4714f4, 0xdf8475800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4002500ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4970, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1080, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1080, 0x40044e3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010608, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001bb560, 0x40044e3000, 0x1000, 0x1000, 0x4000412d80, 0x200000003, 0x4000412d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004feae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004feae0, 0x400043800a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x4000438000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40004feae0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000438000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002507aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002507aa0, 0x4002574000, 0x1000, 0x1000, 0x40000a19b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400046ce00, 0x4002574000, 0x1000, 0x1000, 0x400033feb8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400046ce00, 0x4002574000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400046ce40, 0x4002574000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x4002574201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400257420a, 0x1, 0x4000132b4b, 0x0, 0x4002574263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400012080a, 0x4000132b4b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002500ee0, 0x400041cba0, 0x24, 0x40000a1f20, 0x400011cdf8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4002500ee0, 0x113075a, 0x7, 0x400011cdf8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4002500ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 120 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004c7a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 114 [select]: net/http.(*persistConn).readLoop(0x40001bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x40001bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 92 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400046c340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 136 [semacquire]: sync.runtime_Semacquire(0x4004be3848) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004be3840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40003fc000, 0x400036d200, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcc4a4969, 0x29, 0xffffcc4a49cb, 0x25, 0x4004bc2d80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c8bd20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcc4a4969, 0x29, 0xffffcc4a49cb, 0x25, 0x4004bc2d80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c8bd20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x400042d190, 0x400042ce60, 0x400036d200, 0x40000b9440, 0x400012c580, 0x0, 0x17f, 0x4000178da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000380380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c8bd20, 0xb, 0x4004c8bd18, 0x6, 0x4004c8bd30, 0xd, 0x4004cd1920, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000380380, 0x4000330000, 0x400036c810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000380380, 0x4000415860, 0x2, 0x2, 0x4000415870, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 105 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40023f7f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4318, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1e80, 0x4004a7d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1e80, 0x4004a7d000, 0x1000, 0x1000, 0x400044ba48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010888, 0x4004a7d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400047efc0, 0x4004a7d000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x4000116230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004aa8240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004aa8240, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40004a8c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004aa8240, 0x0, 0x40004a8c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023f5fb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023f5fb0, 0x4004b81800, 0x200, 0x200, 0x4, 0x0, 0x40004a8d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004415140, 0x4004b81800, 0x200, 0x200, 0x40023f8987, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004415140, 0x4004b81800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004415180, 0x4004b81800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b8be40, 0x0, 0x40004a8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b8be40, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b8be40, 0xee0d40, 0x400441e640, 0x12db068, 0x4004415180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bc0750, 0x4004425980, 0x4004bc2660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 131 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400046d3c0, 0x400007f500, 0x400007f570, 0x4000414070, 0x12e8f90, 0x40000e89a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c8ad94, 0x5, 0x4004c8c4c0, 0x12eca70, 0x4004c8c280, 0x130d478, 0x40001f68c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 95 [select]: net/http.(*persistConn).readLoop(0x400047efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [select]: net/http.(*persistConn).writeLoop(0x400047efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 121 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004c7a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 122 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400046d3c0, 0x4000414070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0